[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:12 dist-test-slave-wvz3 krb5kdc[11598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:12 dist-test-slave-wvz3 krb5kdc[11598](info): set up 2 sockets
Jun 20 07:03:12 dist-test-slave-wvz3 krb5kdc[11598](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 07:03:13.624099 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:13 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:13.793794 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38091
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:14.112380 11611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:14.112715 11611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:14.112839 11611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:14.120290 11611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:14.120411 11611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:14.120469 11611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:14.120558 11611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:14.120611 11611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:14.130853 11611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:14.132297 11611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:14.133682 11611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:14.146332 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:14.146466 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:14.146492 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:14.147035 11611 server_base.cc:1034] running on GCE node
I20260620 07:03:14.147938 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:14.149487 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:14.150704 11611 hybrid_clock.cc:648] HybridClock initialized: now 1781938994150646 us; error 46 us; skew 500 ppm
I20260620 07:03:14.151085 11611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:14 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938994, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:14.154345 11611 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:14.155913 11611 webserver.cc:460] Webserver started at http://127.11.81.254:45979/ using document root <none> and password file <none>
I20260620 07:03:14.156622 11611 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:14.156725 11611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:14.157016 11611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:14.160051 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "fe619bf1f327444f9d46bb816197b175"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "84aeb88209160a12a1a6bcfe89826b48"
server_key_iv: "c6a9e45211b21327129c867474ca61d6"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.160828 11611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "fe619bf1f327444f9d46bb816197b175"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "84aeb88209160a12a1a6bcfe89826b48"
server_key_iv: "c6a9e45211b21327129c867474ca61d6"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.166184 11611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:03:14.169878 11626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:14.171446 11611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:03:14.171630 11611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "fe619bf1f327444f9d46bb816197b175"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "84aeb88209160a12a1a6bcfe89826b48"
server_key_iv: "c6a9e45211b21327129c867474ca61d6"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.171828 11611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:14.198406 11611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:14.203509 11611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:14.203804 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:14.222146 11611 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:38091
I20260620 07:03:14.222169 11677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:38091 every 8 connection(s)
I20260620 07:03:14.223569 11611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:14.227582 11678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:14.230906 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 11611
I20260620 07:03:14.231331 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:14.234083 11591 external_mini_cluster.cc:1428] Setting key ae8492a8233c20388b8c96d4a3a84162
I20260620 07:03:14.241406 11678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175: Bootstrap starting.
I20260620 07:03:14.244603 11678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:14.245692 11678 log.cc:824] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:14.248648 11678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175: No bootstrap required, opened a new log
I20260620 07:03:14.257016 11678 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } }
I20260620 07:03:14.257408 11678 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:14.257508 11678 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe619bf1f327444f9d46bb816197b175, State: Initialized, Role: FOLLOWER
I20260620 07:03:14.258108 11678 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } }
I20260620 07:03:14.258337 11678 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:14.258461 11678 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:14.258637 11678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:14.261258 11678 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } }
I20260620 07:03:14.261812 11678 leader_election.cc:302] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe619bf1f327444f9d46bb816197b175; no voters:
I20260620 07:03:14.262457 11678 leader_election.cc:288] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:14.262570 11683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:14.263712 11683 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [term 1 LEADER]: Becoming Leader. State: Replica: fe619bf1f327444f9d46bb816197b175, State: Running, Role: LEADER
Jun 20 07:03:14 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:14.264343 11683 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } }
I20260620 07:03:14.264995 11678 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:14.267534 11685 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe619bf1f327444f9d46bb816197b175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } } }
I20260620 07:03:14.267601 11684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe619bf1f327444f9d46bb816197b175. Latest consensus state: current_term: 1 leader_uuid: "fe619bf1f327444f9d46bb816197b175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe619bf1f327444f9d46bb816197b175" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38091 } } }
I20260620 07:03:14.268005 11684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:14.268002 11685 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:14.268991 11689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:14.279489 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:14.239275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.11.81.254:38091)
0620 07:03:14.240253 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:14.240271 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:14.241674 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 07:03:14.250708 (+ 9034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:14.250742 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:14.251211 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:14.251709 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:14.254772 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.256254 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.258159 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.258986 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.259190 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:14.265534 (+ 6344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:14.265543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:14.265559 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:14.265605 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:14.270283 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:14.271604 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:14.271614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:14.271625 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:14.271885 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:14.272652 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:14.272658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:14.272661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:14.273455 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:14.277025 (+ 3570us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:14.277353 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 07:03:14.277883 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":381,"threads_started":1}
I20260620 07:03:14.285893 11689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:14.296523 11689 catalog_manager.cc:1347] Generated new cluster ID: 7cc6962ecfe047b7abd7602d9b94f246
I20260620 07:03:14.296711 11689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:14.307744 11689 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:14.309128 11689 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:14.316164 11689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe619bf1f327444f9d46bb816197b175: Generated new TSK 0
I20260620 07:03:14.317174 11689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:14.367702 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:14.698756 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:14.699088 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:14.699223 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:14.706640 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:14.706782 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:14.706935 11704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:14.717092 11704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:14.718402 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:14.719936 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:14.733163 11709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:14.733209 11710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:14.733314 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:14.734166 11704 server_base.cc:1034] running on GCE node
I20260620 07:03:14.734802 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:14.735845 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:14.737088 11704 hybrid_clock.cc:648] HybridClock initialized: now 1781938994737021 us; error 54 us; skew 500 ppm
I20260620 07:03:14.737430 11704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:14 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938994, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:14.740780 11704 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:14.742357 11704 webserver.cc:460] Webserver started at http://127.11.81.193:41991/ using document root <none> and password file <none>
I20260620 07:03:14.743068 11704 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:14.743175 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:14.743511 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:14.746507 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "54d79f36d35c4dceb93ba610296228ce"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "7007ff23b6dd100db0edb5c23f6a8b1b"
server_key_iv: "c50f0699e5bb102e814ff5c327f2f094"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.747328 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "54d79f36d35c4dceb93ba610296228ce"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "7007ff23b6dd100db0edb5c23f6a8b1b"
server_key_iv: "c50f0699e5bb102e814ff5c327f2f094"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.752722 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:14.756317 11719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:14.757871 11704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:03:14.758045 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "54d79f36d35c4dceb93ba610296228ce"
format_stamp: "Formatted at 2026-06-20 07:03:14 on dist-test-slave-wvz3"
server_key: "7007ff23b6dd100db0edb5c23f6a8b1b"
server_key_iv: "c50f0699e5bb102e814ff5c327f2f094"
server_key_version: "encryptionkey@0"
I20260620 07:03:14.758242 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:14.788902 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:14.793354 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:14.793649 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:14.794608 11704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:14.796006 11704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:14.796079 11704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:14.796192 11704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:14.796244 11704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:14.825903 11704 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39279
I20260620 07:03:14.825942 11831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39279 every 8 connection(s)
I20260620 07:03:14.827215 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:14.836129 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 11704
I20260620 07:03:14.836349 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:14.836911 11591 external_mini_cluster.cc:1428] Setting key 5a2dd5099cf73a279ac79fe81540a131
Jun 20 07:03:14 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938994, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:14.847803 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:14.829978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:40137 (local address 127.11.81.254:38091)
0620 07:03:14.830446 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:14.830451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:14.831485 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 07:03:14.833404 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:14.833410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:14.833506 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:14.833600 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:14.835278 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.836038 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.837725 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.838252 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.838431 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:14.841949 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:14.841956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:14.841961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:14.842000 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:14.844546 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:14.845313 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:14.845319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:14.845323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:14.845467 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:14.845919 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:14.845924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:14.845928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:14.846271 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:14.846410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:14.847206 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 07:03:14.847451 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260620 07:03:14.848766 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:14.830599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38091 (local address 127.11.81.193:40137)
0620 07:03:14.831326 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:14.831366 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:14.833104 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:14.833869 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:14.833878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:14.834482 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:14.835080 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:14.835097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.836196 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.836200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:14.837575 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:14.837588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:14.838504 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:14.838510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:14.838816 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:14.839515 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:14.839595 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:14.841710 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:14.844704 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:14.844712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:14.844726 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:14.845175 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:14.845569 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:14.845573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:14.845575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:14.845818 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:14.846423 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:14.846429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:14.846831 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 07:03:14.847079 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 07:03:14.847570 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":248,"threads_started":1}
I20260620 07:03:14.850435 11832 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38091
I20260620 07:03:14.850806 11832 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:14.851794 11832 heartbeater.cc:511] Master 127.11.81.254:38091 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:14.854218 11643 ts_manager.cc:194] Registered new tserver with Master: 54d79f36d35c4dceb93ba610296228ce (127.11.81.193:39279)
I20260620 07:03:14.855818 11643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:40137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:14.878564 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:15.205210 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:15.205545 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:15.205670 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:15.213153 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:15.213363 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:15.213584 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:15.224282 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:15.225566 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:15.226873 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:15.240134 11846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:15.240058 11843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:15.240152 11844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:15.240907 11838 server_base.cc:1034] running on GCE node
I20260620 07:03:15.241554 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:15.242587 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:15.243810 11838 hybrid_clock.cc:648] HybridClock initialized: now 1781938995243759 us; error 39 us; skew 500 ppm
I20260620 07:03:15.244172 11838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:15.247534 11838 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:15.249106 11838 webserver.cc:460] Webserver started at http://127.11.81.194:33147/ using document root <none> and password file <none>
I20260620 07:03:15.249809 11838 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:15.249928 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:15.250242 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:15.253382 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "58a0102a072747ab8a0c7dec8b274fce"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "099a47210928ef6c2341b3566ae4c169"
server_key_iv: "a3c43a6cd0e769946902e9df2c548b06"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.254171 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "58a0102a072747ab8a0c7dec8b274fce"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "099a47210928ef6c2341b3566ae4c169"
server_key_iv: "a3c43a6cd0e769946902e9df2c548b06"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.259745 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:03:15.263760 11853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.265251 11838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:03:15.265455 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "58a0102a072747ab8a0c7dec8b274fce"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "099a47210928ef6c2341b3566ae4c169"
server_key_iv: "a3c43a6cd0e769946902e9df2c548b06"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.265659 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:15.283807 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:15.288322 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:15.288580 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:15.289563 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:15.290967 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:15.291043 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.291143 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:15.291188 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.321841 11838 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36797
I20260620 07:03:15.321918 11965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36797 every 8 connection(s)
I20260620 07:03:15.323107 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:15.325270 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 11838
I20260620 07:03:15.325438 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:15.325924 11591 external_mini_cluster.cc:1428] Setting key 23b06d0b2302c546096b997c40ceeb43
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
I20260620 07:03:15.345908 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.325718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:40995 (local address 127.11.81.254:38091)
0620 07:03:15.325924 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:15.325931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:15.327279 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 07:03:15.329171 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:15.329178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:15.329263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:15.329376 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:15.331037 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.332004 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.334330 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.334796 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.334898 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.338922 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:15.338930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:15.338935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:15.338983 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:15.342700 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.343556 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.343561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.343565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.343731 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.344119 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.344122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.344124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.344409 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:15.344519 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:15.345341 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 07:03:15.345581 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:03:15.346951 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.326297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38091 (local address 127.11.81.194:40995)
0620 07:03:15.327107 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:15.327155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:15.328896 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:15.329639 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:15.329648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:15.330256 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:15.330861 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:15.330881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.332163 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.332169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:15.334170 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:15.334184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.335018 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.335024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:15.335355 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.336041 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:15.336073 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:15.338661 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:15.342880 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:15.342890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:15.342907 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:15.343438 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:15.343835 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:15.343839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:15.343841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:15.344029 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:15.344521 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:15.344529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:15.344935 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 07:03:15.345226 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 07:03:15.345783 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":319,"threads_started":1}
I20260620 07:03:15.348526 11966 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38091
I20260620 07:03:15.348872 11966 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:15.349785 11966 heartbeater.cc:511] Master 127.11.81.254:38091 requested a full tablet report, sending...
I20260620 07:03:15.351491 11643 ts_manager.cc:194] Registered new tserver with Master: 58a0102a072747ab8a0c7dec8b274fce (127.11.81.194:36797)
I20260620 07:03:15.352383 11643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:40995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:15.367444 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:15.689990 11971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:15.690315 11971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:15.690415 11971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:15.697849 11971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:15.697980 11971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:15.698107 11971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:15.708122 11971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38091
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:15.709383 11971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:15.710649 11971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:15.723670 11977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:15.723703 11976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:15.723631 11979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:15.724805 11971 server_base.cc:1034] running on GCE node
I20260620 07:03:15.725410 11971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:15.726329 11971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:15.727515 11971 hybrid_clock.cc:648] HybridClock initialized: now 1781938995727450 us; error 54 us; skew 500 ppm
I20260620 07:03:15.727835 11971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:15.731127 11971 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:15.732717 11971 webserver.cc:460] Webserver started at http://127.11.81.195:39617/ using document root <none> and password file <none>
I20260620 07:03:15.733394 11971 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:15.733496 11971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:15.733804 11971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:15.736804 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "938c82e67b6348b0955c631ecab6fac5"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "4279d12dbd6654fda028477c33bab608"
server_key_iv: "af7885f68cac9fdf8db33d261d6861b2"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.737545 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "938c82e67b6348b0955c631ecab6fac5"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "4279d12dbd6654fda028477c33bab608"
server_key_iv: "af7885f68cac9fdf8db33d261d6861b2"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.742718 11971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:03:15.746344 11986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.747826 11971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:15.747982 11971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "938c82e67b6348b0955c631ecab6fac5"
format_stamp: "Formatted at 2026-06-20 07:03:15 on dist-test-slave-wvz3"
server_key: "4279d12dbd6654fda028477c33bab608"
server_key_iv: "af7885f68cac9fdf8db33d261d6861b2"
server_key_version: "encryptionkey@0"
I20260620 07:03:15.748164 11971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:15.767009 11971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:15.771438 11971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:15.771725 11971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:15.772706 11971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:15.774040 11971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:15.774120 11971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.774240 11971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:15.774291 11971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:15.804555 11971 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36143
I20260620 07:03:15.804639 12098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36143 every 8 connection(s)
I20260620 07:03:15.805850 11971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:15.814532 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 11971
I20260620 07:03:15.814733 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:15.815162 11591 external_mini_cluster.cc:1428] Setting key 6853fb07974c7ed78a026d5619909c22
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:15.826584 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.808475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:46685 (local address 127.11.81.254:38091)
0620 07:03:15.808685 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:15.808689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:15.809840 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 07:03:15.811726 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:15.811731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:15.811834 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:15.811929 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:15.813526 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.814470 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.816331 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.816840 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.816974 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.820914 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:15.820919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:15.820923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:15.820959 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:15.823566 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.824300 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.824304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.824306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.824435 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.824860 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.824863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.824865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.825166 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:15.825303 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:15.826023 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 07:03:15.826261 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:03:15.827608 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.808943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38091 (local address 127.11.81.195:46685)
0620 07:03:15.809673 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:15.809711 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:15.811432 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:15.812186 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:15.812194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:15.812749 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:15.813352 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:15.813371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.814647 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.814653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:15.816196 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:15.816209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.817075 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.817081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:15.817331 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.818089 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:15.818127 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:15.820699 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:15.823717 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:15.823726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:15.823742 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:15.824165 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:15.824540 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:15.824545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:15.824548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:15.824744 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:15.825297 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:15.825304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:15.825646 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 07:03:15.825895 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:03:15.826433 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":247,"threads_started":1}
I20260620 07:03:15.829332 12099 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38091
I20260620 07:03:15.829654 12099 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:15.830556 12099 heartbeater.cc:511] Master 127.11.81.254:38091 requested a full tablet report, sending...
I20260620 07:03:15.832080 11643 ts_manager.cc:194] Registered new tserver with Master: 938c82e67b6348b0955c631ecab6fac5 (127.11.81.195:36143)
I20260620 07:03:15.832921 11643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:46685
I20260620 07:03:15.844494 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:15.858848 11832 heartbeater.cc:503] Master 127.11.81.254:38091 was elected leader, sending a full tablet report...
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:15.887776 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.872300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.11.81.193:39279)
0620 07:03:15.872935 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:15.872944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:15.872977 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:03:15.873163 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:15.873177 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:15.873414 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:15.873561 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:15.874984 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.876099 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.877434 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.878004 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.878125 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.882019 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:15.882033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:15.882049 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:15.882090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:15.884931 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.885635 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.885642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.885645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.885774 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.886136 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.886140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.886142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.886507 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:15.886795 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:15.887106 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:03:15.887429 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
W20260620 07:03:15.889113 11747 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51248
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:03:15 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:15.931548 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.917121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.11.81.193:39279)
0620 07:03:15.917349 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:15.917355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:15.917691 (+ 336us) server_negotiation.cc:407] Connection header received
0620 07:03:15.917908 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:15.917912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:15.918012 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:15.918096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:15.919478 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.920242 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.921495 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.922017 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.922144 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.926216 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:15.926221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:15.926225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:15.926265 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:15.928883 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.929618 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.929624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.929627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.929781 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.930147 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.930151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.930153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.930421 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:15.930646 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:15.930969 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 07:03:15.931206 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:03:15.933171 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 11704
I20260620 07:03:15.946128 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:15.934034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.11.81.193:39279)
0620 07:03:15.934196 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:15.934201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:15.934693 (+ 492us) server_negotiation.cc:407] Connection header received
0620 07:03:15.934788 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:15.934792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:15.934858 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:15.934963 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:15.936264 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.936952 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.938199 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:15.938619 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:15.938729 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:15.940998 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:15.941003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:15.941006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:15.941037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:15.943550 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.944280 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.944286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.944290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.944428 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:15.944789 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:15.944793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:15.944795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:15.945055 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:15.945286 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:15.945569 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 07:03:15.945781 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:03:15.946817 11827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:16.030089 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 11704
I20260620 07:03:16.042965 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 11838
Jun 20 07:03:16 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:16.059705 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:16.043947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.11.81.194:36797)
0620 07:03:16.044488 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:16.044495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:16.044520 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:16.044586 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:16.044595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:16.044797 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:16.044928 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:16.046012 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:16.047037 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:16.048271 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:16.048826 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:16.048926 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:16.053149 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:16.053164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:16.053179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:16.053219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:16.056575 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:16.057445 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:16.057453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:16.057456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:16.057599 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:16.057981 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:16.057985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:16.057987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:16.058364 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:16.058657 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:16.058964 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:03:16.059333 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
I20260620 07:03:16.061056 11961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:16.143321 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 11838
I20260620 07:03:16.155736 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 11971
Jun 20 07:03:16 dist-test-slave-wvz3 krb5kdc[11598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:16.171859 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:16.156645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.11.81.195:36143)
0620 07:03:16.157269 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:16.157276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:16.157304 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:03:16.157364 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:16.157371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:16.157594 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:16.157721 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:16.158892 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:16.159993 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:16.161371 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:16.161961 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:16.162072 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:16.166198 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:16.166211 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:16.166227 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:16.166265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:16.169220 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:16.169934 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:16.169941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:16.169944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:16.170069 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:16.170336 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:16.170339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:16.170342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:16.170668 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:16.170897 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:16.171170 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 07:03:16.171505 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":312,"threads_started":1}
I20260620 07:03:16.173171 12094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:16.255659 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 11971
I20260620 07:03:16.268201 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 11611
I20260620 07:03:16.269389 11673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:16.374120 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 11611
2026-06-20T07:03:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3763 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:16 dist-test-slave-wvz3 krb5kdc[12120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:16 dist-test-slave-wvz3 krb5kdc[12120](info): set up 2 sockets
Jun 20 07:03:16 dist-test-slave-wvz3 krb5kdc[12120](info): commencing operation
krb5kdc: starting...
W20260620 07:03:17.359097 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:17 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:17.546763 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40521
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:17.861518 12133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:17.861824 12133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:17.861959 12133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:17.869251 12133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:17.869390 12133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:17.869473 12133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:17.869546 12133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:17.869607 12133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:17.879570 12133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:17.880903 12133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:17.882292 12133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:17.894438 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:17.894397 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:17.894433 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:17.895236 12133 server_base.cc:1034] running on GCE node
I20260620 07:03:17.896123 12133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:17.897631 12133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:17.898854 12133 hybrid_clock.cc:648] HybridClock initialized: now 1781938997898801 us; error 42 us; skew 500 ppm
I20260620 07:03:17.899237 12133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:17 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938997, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:17.907407 12133 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:17.909229 12133 webserver.cc:460] Webserver started at http://127.11.81.254:32889/ using document root <none> and password file <none>
I20260620 07:03:17.909992 12133 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:17.910125 12133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:17.910440 12133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:17.913601 12133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "3ebaa59c7a3f4871a629abb454d5dad1"
format_stamp: "Formatted at 2026-06-20 07:03:17 on dist-test-slave-wvz3"
server_key: "ff5d00aa2f9caf456cede7e4f0ea9462"
server_key_iv: "bbf370aabe9ba596faf164ddf33f1b69"
server_key_version: "encryptionkey@0"
I20260620 07:03:17.914402 12133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "3ebaa59c7a3f4871a629abb454d5dad1"
format_stamp: "Formatted at 2026-06-20 07:03:17 on dist-test-slave-wvz3"
server_key: "ff5d00aa2f9caf456cede7e4f0ea9462"
server_key_iv: "bbf370aabe9ba596faf164ddf33f1b69"
server_key_version: "encryptionkey@0"
I20260620 07:03:17.920095 12133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:03:17.924122 12148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:17.925657 12133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:17.925829 12133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "3ebaa59c7a3f4871a629abb454d5dad1"
format_stamp: "Formatted at 2026-06-20 07:03:17 on dist-test-slave-wvz3"
server_key: "ff5d00aa2f9caf456cede7e4f0ea9462"
server_key_iv: "bbf370aabe9ba596faf164ddf33f1b69"
server_key_version: "encryptionkey@0"
I20260620 07:03:17.926031 12133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:17.956113 12133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:17.960626 12133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:17.960870 12133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:17.979499 12133 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:40521
I20260620 07:03:17.979471 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:40521 every 8 connection(s)
I20260620 07:03:17.981078 12133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:17.983644 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12133
I20260620 07:03:17.983880 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:17.984540 11591 external_mini_cluster.cc:1428] Setting key d5772a8005b6856f46c7cdcedac0be48
I20260620 07:03:17.985255 12200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:17.999583 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1: Bootstrap starting.
Jun 20 07:03:17 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:18.003535 12200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:18.004969 12200 log.cc:824] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:18.008736 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1: No bootstrap required, opened a new log
I20260620 07:03:18.009959 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:17.986669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.11.81.254:40521)
0620 07:03:17.987537 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:17.987552 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:17.987602 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:03:17.990025 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:17.990058 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:17.990628 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:17.991105 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:17.992494 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:17.994432 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:17.995799 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:17.996938 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:17.997136 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:18.001017 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:18.001038 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:18.001064 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:18.001125 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:18.005216 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:18.006040 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:18.006061 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:18.006074 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:18.006297 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:18.006843 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:18.006847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:18.006849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:18.007588 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:18.007856 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:18.008462 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 07:03:18.008919 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":331,"threads_started":1}
I20260620 07:03:18.016994 12200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } }
I20260620 07:03:18.017387 12200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:18.017484 12200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebaa59c7a3f4871a629abb454d5dad1, State: Initialized, Role: FOLLOWER
I20260620 07:03:18.018074 12200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } }
I20260620 07:03:18.018288 12200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:18.018409 12200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:18.018573 12200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:18.020881 12200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } }
I20260620 07:03:18.021458 12200 leader_election.cc:302] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ebaa59c7a3f4871a629abb454d5dad1; no voters:
I20260620 07:03:18.022344 12200 leader_election.cc:288] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:18.022435 12205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:18.023099 12205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ebaa59c7a3f4871a629abb454d5dad1, State: Running, Role: LEADER
I20260620 07:03:18.023741 12205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } }
I20260620 07:03:18.024538 12200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:18.026649 12207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } } }
I20260620 07:03:18.027097 12207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:18.029755 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ebaa59c7a3f4871a629abb454d5dad1. Latest consensus state: current_term: 1 leader_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebaa59c7a3f4871a629abb454d5dad1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40521 } } }
I20260620 07:03:18.030231 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:18.031724 12212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:18.036468 12212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:18.046058 12212 catalog_manager.cc:1347] Generated new cluster ID: cc76a6d2721a4110b3c4e12e9b2f8bcf
I20260620 07:03:18.046201 12212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:18.056572 12212 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:18.057896 12212 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:18.063963 12212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ebaa59c7a3f4871a629abb454d5dad1: Generated new TSK 0
I20260620 07:03:18.065088 12212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:18.122498 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:18.445511 12226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:18.445801 12226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:18.445897 12226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:18.453225 12226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:18.453348 12226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:18.453464 12226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:18.463613 12226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:18.464900 12226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:18.466225 12226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:18.479770 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:18.479887 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:18.479923 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:18.480671 12226 server_base.cc:1034] running on GCE node
I20260620 07:03:18.481392 12226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:18.482609 12226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:18.483850 12226 hybrid_clock.cc:648] HybridClock initialized: now 1781938998483775 us; error 65 us; skew 500 ppm
I20260620 07:03:18.484195 12226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:18 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938998, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:18.489653 12226 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:18.491215 12226 webserver.cc:460] Webserver started at http://127.11.81.193:37721/ using document root <none> and password file <none>
I20260620 07:03:18.491925 12226 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:18.492030 12226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:18.492339 12226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:18.495270 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "146299b0fdf74f4a9b52ab72aece32c3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "e12a3fcb64f9718ebd6f69ba100b231d"
server_key_iv: "14cf57a574afdae8ec2588d68a066d78"
server_key_version: "encryptionkey@0"
I20260620 07:03:18.496105 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "146299b0fdf74f4a9b52ab72aece32c3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "e12a3fcb64f9718ebd6f69ba100b231d"
server_key_iv: "14cf57a574afdae8ec2588d68a066d78"
server_key_version: "encryptionkey@0"
I20260620 07:03:18.501281 12226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:18.504915 12241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:18.506394 12226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 07:03:18.506557 12226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "146299b0fdf74f4a9b52ab72aece32c3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "e12a3fcb64f9718ebd6f69ba100b231d"
server_key_iv: "14cf57a574afdae8ec2588d68a066d78"
server_key_version: "encryptionkey@0"
I20260620 07:03:18.506800 12226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:18.548156 12226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:18.552649 12226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:18.552946 12226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:18.553910 12226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:18.555267 12226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:18.555366 12226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:18.555485 12226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:18.555538 12226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:18.586916 12226 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:43405
I20260620 07:03:18.586933 12353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:43405 every 8 connection(s)
I20260620 07:03:18.588276 12226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:18.590813 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12226
I20260620 07:03:18.591029 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:18.591815 11591 external_mini_cluster.cc:1428] Setting key cb0015e14ed35ba4974543903a210937
Jun 20 07:03:18 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938998, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:18.610662 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:18.590857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:54375 (local address 127.11.81.254:40521)
0620 07:03:18.591427 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:18.591432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:18.592727 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 07:03:18.594900 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:18.594907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:18.595024 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:18.595129 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:18.596889 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:18.597653 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:18.599273 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:18.599976 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:18.600117 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:18.603916 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:18.603925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:18.603931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:18.603979 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:18.607353 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:18.608153 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:18.608161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:18.608165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:18.608332 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:18.608801 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:18.608805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:18.608808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:18.609138 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:18.609267 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:18.610027 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 07:03:18.610285 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:03:18.611935 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:18.591643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40521 (local address 127.11.81.193:54375)
0620 07:03:18.592518 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:18.592574 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:18.594606 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:18.595468 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:18.595478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:18.596040 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:18.596687 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:18.596709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:18.597813 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:18.597817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:18.599114 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:18.599128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:18.600213 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:18.600218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:18.600424 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:18.601143 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:18.601173 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:18.603644 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:18.607515 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:18.607523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:18.607537 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:18.607982 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:18.608442 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:18.608445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:18.608448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:18.608660 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:18.609261 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:18.609267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:18.609601 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 07:03:18.609847 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 07:03:18.610438 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":358,"threads_started":1}
I20260620 07:03:18.617259 12354 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40521
I20260620 07:03:18.617676 12354 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:18.618685 12354 heartbeater.cc:511] Master 127.11.81.254:40521 requested a full tablet report, sending...
I20260620 07:03:18.621532 12165 ts_manager.cc:194] Registered new tserver with Master: 146299b0fdf74f4a9b52ab72aece32c3 (127.11.81.193:43405)
I20260620 07:03:18.623983 12165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:54375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:18.634337 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:18.951061 12360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:18.951381 12360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:18.951479 12360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:18.958673 12360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:18.958832 12360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:18.958971 12360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:18.968894 12360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:18.970175 12360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:18.971489 12360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:18.984586 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:18.984984 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:18.984640 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:18.985749 12360 server_base.cc:1034] running on GCE node
I20260620 07:03:18.986498 12360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:18.987664 12360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:18.988890 12360 hybrid_clock.cc:648] HybridClock initialized: now 1781938998988828 us; error 47 us; skew 500 ppm
I20260620 07:03:18.989265 12360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:18 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938998, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:18.993233 12360 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:18.994937 12360 webserver.cc:460] Webserver started at http://127.11.81.194:33845/ using document root <none> and password file <none>
I20260620 07:03:18.995831 12360 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:18.995954 12360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:18.996307 12360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:18.999512 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "0b17bb957d86454582af6a379307dcb3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "eec5359df65a82bda303320b67527d99"
server_key_iv: "e081aa86dcc6c057a93b5d45c6388727"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.000294 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "0b17bb957d86454582af6a379307dcb3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "eec5359df65a82bda303320b67527d99"
server_key_iv: "e081aa86dcc6c057a93b5d45c6388727"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.005502 12360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:03:19.009212 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.010703 12360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:03:19.010854 12360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "0b17bb957d86454582af6a379307dcb3"
format_stamp: "Formatted at 2026-06-20 07:03:18 on dist-test-slave-wvz3"
server_key: "eec5359df65a82bda303320b67527d99"
server_key_iv: "e081aa86dcc6c057a93b5d45c6388727"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.011030 12360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:19.038429 12360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:19.042714 12360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:19.042958 12360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:19.043938 12360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:19.045341 12360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:19.045413 12360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.045497 12360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:19.045538 12360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.076078 12360 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:45885
I20260620 07:03:19.076144 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:45885 every 8 connection(s)
I20260620 07:03:19.077309 12360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:19.081280 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12360
I20260620 07:03:19.081513 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:19.081948 11591 external_mini_cluster.cc:1428] Setting key c4ef1fb7dc70a897892918214d7857b3
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938998, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:03:19.100991 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.079945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:59941 (local address 127.11.81.254:40521)
0620 07:03:19.080109 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.080114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.081326 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 07:03:19.083986 (+ 2660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.083991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.084069 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.084157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.086234 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.087369 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.089381 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.090000 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.090141 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.094125 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.094130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.094133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.094171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.097422 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.098445 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.098449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.098452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.098591 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.099156 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.099160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.099162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.099548 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.099691 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.100482 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.100731 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
I20260620 07:03:19.102181 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.080381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40521 (local address 127.11.81.194:59941)
0620 07:03:19.081142 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.081187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.083698 (+ 2511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.084452 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.084464 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.085204 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:19.086051 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.086078 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.087515 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.087521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.089214 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.089232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.090257 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.090262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.090470 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.091267 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:19.091316 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:19.093813 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:19.097592 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:19.097602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:19.097623 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:19.098315 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:19.098717 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:19.098722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:19.098726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:19.098992 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:19.099686 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:19.099692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:19.100063 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.100343 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.100796 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":308,"threads_started":1}
I20260620 07:03:19.103737 12488 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40521
I20260620 07:03:19.104102 12488 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:19.104990 12488 heartbeater.cc:511] Master 127.11.81.254:40521 requested a full tablet report, sending...
I20260620 07:03:19.106845 12165 ts_manager.cc:194] Registered new tserver with Master: 0b17bb957d86454582af6a379307dcb3 (127.11.81.194:45885)
I20260620 07:03:19.107867 12165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:59941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:19.127368 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:19.446022 12493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:19.446342 12493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:19.446480 12493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:19.453727 12493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:19.453861 12493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:19.454015 12493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:19.464032 12493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:19.465349 12493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:19.466758 12493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:19.480867 12498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:19.481025 12499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:19.482299 12501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:19.483696 12493 server_base.cc:1034] running on GCE node
I20260620 07:03:19.484498 12493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:19.485656 12493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:19.486883 12493 hybrid_clock.cc:648] HybridClock initialized: now 1781938999486817 us; error 60 us; skew 500 ppm
I20260620 07:03:19.487215 12493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:19.490490 12493 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:19.492170 12493 webserver.cc:460] Webserver started at http://127.11.81.195:40337/ using document root <none> and password file <none>
I20260620 07:03:19.492875 12493 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:19.492985 12493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:19.493299 12493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:19.496308 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "b570f95970e449b0936c9fb766520bbc"
format_stamp: "Formatted at 2026-06-20 07:03:19 on dist-test-slave-wvz3"
server_key: "3aee770b07cf96523402c5699228d84a"
server_key_iv: "096cfe8e92e7a39f6605b0df2443e861"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.497069 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "b570f95970e449b0936c9fb766520bbc"
format_stamp: "Formatted at 2026-06-20 07:03:19 on dist-test-slave-wvz3"
server_key: "3aee770b07cf96523402c5699228d84a"
server_key_iv: "096cfe8e92e7a39f6605b0df2443e861"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.502301 12493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 07:03:19.505995 12508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.507604 12493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:19.507776 12493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b570f95970e449b0936c9fb766520bbc"
format_stamp: "Formatted at 2026-06-20 07:03:19 on dist-test-slave-wvz3"
server_key: "3aee770b07cf96523402c5699228d84a"
server_key_iv: "096cfe8e92e7a39f6605b0df2443e861"
server_key_version: "encryptionkey@0"
I20260620 07:03:19.507987 12493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:19.535312 12493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:19.539911 12493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:19.540197 12493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:19.541240 12493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:19.542716 12493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:19.542795 12493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.542917 12493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:19.542969 12493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:19.573954 12493 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:44315
I20260620 07:03:19.574002 12620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:44315 every 8 connection(s)
I20260620 07:03:19.575332 12493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:19.585093 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12493
I20260620 07:03:19.585333 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:19.585857 11591 external_mini_cluster.cc:1428] Setting key 10c45d212de5bc781e28ef43b802f260
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:19.597067 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.578269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:47623 (local address 127.11.81.254:40521)
0620 07:03:19.578464 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.578469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.579875 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 07:03:19.581837 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.581842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.581958 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.582080 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.583826 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.584630 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.587002 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.587680 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.587826 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.591666 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.591673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.591678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.591721 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.594229 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.594990 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.594994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.594997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.595139 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.595569 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.595574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.595576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.595853 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.595961 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.596619 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.596802 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:03:19.598070 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.578874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40521 (local address 127.11.81.195:47623)
0620 07:03:19.579697 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.579736 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.581496 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.582336 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.582346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.582926 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:19.583620 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.583641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.584843 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.584849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.586817 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.586841 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.587933 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.587942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.588224 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.588916 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:19.588950 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:19.591440 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:19.594366 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:19.594374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:19.594387 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:19.594866 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:19.595233 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:19.595236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:19.595238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:19.595468 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:19.595955 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:19.595960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:19.596285 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.596512 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.596956 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":348,"threads_started":1}
I20260620 07:03:19.599607 12621 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40521
I20260620 07:03:19.599947 12621 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:19.600785 12621 heartbeater.cc:511] Master 127.11.81.254:40521 requested a full tablet report, sending...
I20260620 07:03:19.602269 12165 ts_manager.cc:194] Registered new tserver with Master: b570f95970e449b0936c9fb766520bbc (127.11.81.195:44315)
I20260620 07:03:19.603107 12165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:47623
I20260620 07:03:19.614342 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:19.628106 12354 heartbeater.cc:503] Master 127.11.81.254:40521 was elected leader, sending a full tablet report...
I20260620 07:03:19.634166 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.621065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.11.81.254:40521)
0620 07:03:19.621263 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.621267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.621659 (+ 392us) server_negotiation.cc:407] Connection header received
0620 07:03:19.621785 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.621789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.621859 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.621953 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.623471 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.624247 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.625596 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.626038 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.626152 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.628888 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.628894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.628899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.628941 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.631788 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.632497 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.632500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.632503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.632645 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.632931 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.632934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.632936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.633265 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.633442 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.633756 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.633913 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:03:19.641004 12165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:19.648000 12165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:19.682726 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.660818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44315 (local address 127.0.0.1:51096)
0620 07:03:19.669289 (+ 8471us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.669306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.669412 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.675411 (+ 5999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.675419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.675455 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:19.675863 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.675877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.678037 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.678042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.679791 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.679809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.681748 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.681755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.681946 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.681999 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.682126 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.682337 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8357,"thread_start_us":7140,"threads_started":1}
I20260620 07:03:19.684055 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.661400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.11.81.195:44315)
0620 07:03:19.674243 (+ 12843us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.674253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.674298 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:03:19.674403 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.674412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.674705 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.676072 (+ 1367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:19.676362 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.677892 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.680013 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.681567 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.681694 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.683412 (+ 1718us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.683493 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.683693 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12609,"thread_start_us":13283,"threads_started":1}
I20260620 07:03:19.684204 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.661237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:43405 (local address 127.0.0.1:33878)
0620 07:03:19.675350 (+ 14113us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.675369 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.675477 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.677210 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.677214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.677240 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:19.677682 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.677694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.679479 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.679483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.680943 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.680959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.683332 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.683340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.683508 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.683552 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.683635 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.683791 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13936,"mutex_wait_us":1,"spinlock_wait_cycles":8960,"thread_start_us":13790,"threads_started":1}
I20260620 07:03:19.684460 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.661741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.11.81.193:43405)
0620 07:03:19.676373 (+ 14632us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.676380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.676405 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:19.676498 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.676504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.676736 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.676951 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:19.677880 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.679343 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.681123 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.682394 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.682526 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.683358 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.683805 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.683976 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14478,"thread_start_us":14328,"threads_started":1}
I20260620 07:03:19.684790 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.660833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.11.81.194:45885)
0620 07:03:19.675944 (+ 15111us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.675953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.675980 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:03:19.676075 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.676082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.676382 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.676646 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:19.677527 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.679061 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.681627 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.683351 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.683488 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.683686 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.684342 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.684515 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14943,"thread_start_us":14847,"threads_started":1}
I20260620 07:03:19.684911 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.660706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45885 (local address 127.0.0.1:38466)
0620 07:03:19.668652 (+ 7946us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.668711 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.668914 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.676657 (+ 7743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.676661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.676689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:19.677072 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.677088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.679222 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.679227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.681447 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.681462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.683564 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.683571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.683744 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.683784 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.683874 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.684005 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7627,"mutex_wait_us":62,"thread_start_us":454,"threads_started":1}
I20260620 07:03:19.687922 12423 tablet_service.cc:1467] Processing CreateTablet for tablet 8f85cbc46f664234abf5b688100ab783 (DEFAULT_TABLE table=test-table [id=c2cd3d73fc7c4dca8e41e11311b2b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:19.688042 12289 tablet_service.cc:1467] Processing CreateTablet for tablet 8f85cbc46f664234abf5b688100ab783 (DEFAULT_TABLE table=test-table [id=c2cd3d73fc7c4dca8e41e11311b2b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:19.688066 12556 tablet_service.cc:1467] Processing CreateTablet for tablet 8f85cbc46f664234abf5b688100ab783 (DEFAULT_TABLE table=test-table [id=c2cd3d73fc7c4dca8e41e11311b2b29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:19.689373 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f85cbc46f664234abf5b688100ab783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:19.689440 12289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f85cbc46f664234abf5b688100ab783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:19.689921 12556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f85cbc46f664234abf5b688100ab783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:19.700757 12639 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Bootstrap starting.
I20260620 07:03:19.704087 12641 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Bootstrap starting.
I20260620 07:03:19.705883 12640 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Bootstrap starting.
I20260620 07:03:19.706396 12639 tablet_bootstrap.cc:654] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:19.707825 12639 log.cc:824] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:19.708114 12641 tablet_bootstrap.cc:654] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:19.708920 12640 tablet_bootstrap.cc:654] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:19.709262 12641 log.cc:824] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:19.710012 12640 log.cc:824] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:19.711196 12639 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: No bootstrap required, opened a new log
I20260620 07:03:19.711520 12639 ts_tablet_manager.cc:1397] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 07:03:19.712670 12641 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: No bootstrap required, opened a new log
I20260620 07:03:19.712934 12641 ts_tablet_manager.cc:1397] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:03:19.723273 12641 raft_consensus.cc:348] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.723266 12639 raft_consensus.cc:348] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.723843 12639 raft_consensus.cc:374] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:19.723843 12641 raft_consensus.cc:374] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:19.723948 12639 raft_consensus.cc:729] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b17bb957d86454582af6a379307dcb3, State: Initialized, Role: FOLLOWER
I20260620 07:03:19.723946 12641 raft_consensus.cc:729] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b570f95970e449b0936c9fb766520bbc, State: Initialized, Role: FOLLOWER
I20260620 07:03:19.724634 12639 consensus_queue.cc:260] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.724633 12641 consensus_queue.cc:260] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.726698 12641 ts_tablet_manager.cc:1428] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 07:03:19.727219 12640 tablet_bootstrap.cc:492] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: No bootstrap required, opened a new log
I20260620 07:03:19.727242 12621 heartbeater.cc:503] Master 127.11.81.254:40521 was elected leader, sending a full tablet report...
I20260620 07:03:19.727604 12640 ts_tablet_manager.cc:1397] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.008s
I20260620 07:03:19.727592 12488 heartbeater.cc:503] Master 127.11.81.254:40521 was elected leader, sending a full tablet report...
I20260620 07:03:19.728992 12639 ts_tablet_manager.cc:1428] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260620 07:03:19.737330 12640 raft_consensus.cc:348] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.737737 12640 raft_consensus.cc:374] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:19.737843 12640 raft_consensus.cc:729] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146299b0fdf74f4a9b52ab72aece32c3, State: Initialized, Role: FOLLOWER
I20260620 07:03:19.738454 12640 consensus_queue.cc:260] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.739997 12640 ts_tablet_manager.cc:1428] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 07:03:19.744275 12647 raft_consensus.cc:482] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:19.744580 12647 raft_consensus.cc:504] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.746546 12647 leader_election.cc:288] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b17bb957d86454582af6a379307dcb3 (127.11.81.194:45885), b570f95970e449b0936c9fb766520bbc (127.11.81.195:44315)
I20260620 07:03:19.754891 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.747024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:41987 (local address 127.11.81.194:45885)
0620 07:03:19.747223 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.747229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.747870 (+ 641us) server_negotiation.cc:407] Connection header received
0620 07:03:19.748107 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.748113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.748196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.748292 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:19.748827 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.749853 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.752482 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.753706 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.753880 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.753958 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.754493 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.754633 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:03:19.754926 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.747108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:36103 (local address 127.11.81.195:44315)
0620 07:03:19.747278 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.747315 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.748376 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 07:03:19.748487 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.748491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.748554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.748619 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:19.749198 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.750219 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.752646 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.753796 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.753929 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.753993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.754489 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.754699 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:03:19.755059 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.746771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45885 (local address 127.11.81.193:41987)
0620 07:03:19.747746 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.747763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.747907 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.748328 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.748333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.748361 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:19.748687 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.748698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.750020 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.750025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.752289 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.752306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.753964 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.753975 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.754201 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.754335 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.754434 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.754614 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":115,"thread_start_us":682,"threads_started":1}
I20260620 07:03:19.755496 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.746815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44315 (local address 127.11.81.193:36103)
0620 07:03:19.748258 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:19.748278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:19.748384 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:19.748679 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:19.748683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:19.748706 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:19.749046 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.749055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.750345 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.750350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.752431 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:19.752443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.753994 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.754001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:19.754241 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.754334 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 07:03:19.754434 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:03:19.754652 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":93,"thread_start_us":1214,"threads_started":1}
I20260620 07:03:19.756121 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f85cbc46f664234abf5b688100ab783" candidate_uuid: "146299b0fdf74f4a9b52ab72aece32c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b17bb957d86454582af6a379307dcb3" is_pre_election: true
I20260620 07:03:19.756388 12576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f85cbc46f664234abf5b688100ab783" candidate_uuid: "146299b0fdf74f4a9b52ab72aece32c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b570f95970e449b0936c9fb766520bbc" is_pre_election: true
I20260620 07:03:19.756629 12443 raft_consensus.cc:2454] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 146299b0fdf74f4a9b52ab72aece32c3 in term 0.
I20260620 07:03:19.756889 12576 raft_consensus.cc:2454] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 146299b0fdf74f4a9b52ab72aece32c3 in term 0.
I20260620 07:03:19.757331 12245 leader_election.cc:302] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b17bb957d86454582af6a379307dcb3, 146299b0fdf74f4a9b52ab72aece32c3; no voters:
I20260620 07:03:19.757881 12647 raft_consensus.cc:2781] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:19.758010 12647 raft_consensus.cc:482] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:19.758132 12647 raft_consensus.cc:3037] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:19.760383 12647 raft_consensus.cc:504] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.761015 12647 leader_election.cc:288] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0b17bb957d86454582af6a379307dcb3 (127.11.81.194:45885), b570f95970e449b0936c9fb766520bbc (127.11.81.195:44315)
I20260620 07:03:19.761626 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f85cbc46f664234abf5b688100ab783" candidate_uuid: "146299b0fdf74f4a9b52ab72aece32c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b17bb957d86454582af6a379307dcb3"
I20260620 07:03:19.761734 12576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f85cbc46f664234abf5b688100ab783" candidate_uuid: "146299b0fdf74f4a9b52ab72aece32c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b570f95970e449b0936c9fb766520bbc"
I20260620 07:03:19.761840 12443 raft_consensus.cc:3037] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:19.761938 12576 raft_consensus.cc:3037] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:19.764251 12443 raft_consensus.cc:2454] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146299b0fdf74f4a9b52ab72aece32c3 in term 1.
I20260620 07:03:19.764262 12576 raft_consensus.cc:2454] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146299b0fdf74f4a9b52ab72aece32c3 in term 1.
I20260620 07:03:19.764770 12242 leader_election.cc:302] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146299b0fdf74f4a9b52ab72aece32c3, b570f95970e449b0936c9fb766520bbc; no voters:
I20260620 07:03:19.765141 12647 raft_consensus.cc:2781] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:19.765657 12647 raft_consensus.cc:686] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [term 1 LEADER]: Becoming Leader. State: Replica: 146299b0fdf74f4a9b52ab72aece32c3, State: Running, Role: LEADER
I20260620 07:03:19.766260 12647 consensus_queue.cc:237] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } }
I20260620 07:03:19.770871 12164 catalog_manager.cc:5495] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 146299b0fdf74f4a9b52ab72aece32c3 (127.11.81.193). New cstate: current_term: 1 leader_uuid: "146299b0fdf74f4a9b52ab72aece32c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b17bb957d86454582af6a379307dcb3" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b570f95970e449b0936c9fb766520bbc" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146299b0fdf74f4a9b52ab72aece32c3" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43405 } health_report { overall_health: HEALTHY } } }
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:19.808065 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.792924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.11.81.193:43405)
0620 07:03:19.793188 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.793194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.793379 (+ 185us) server_negotiation.cc:407] Connection header received
0620 07:03:19.793518 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.793523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.793630 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.793713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.795165 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.795915 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.797286 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.797705 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.797799 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.802155 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.802173 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.802189 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.802236 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.805155 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.805923 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.805931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.805933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.806092 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.806475 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.806478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.806481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.806847 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.807130 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.807490 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.807753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 07:03:19.827035 12622 tablet.cc:2369] T 8f85cbc46f664234abf5b688100ab783 P b570f95970e449b0936c9fb766520bbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:19.829205 12489 tablet.cc:2369] T 8f85cbc46f664234abf5b688100ab783 P 0b17bb957d86454582af6a379307dcb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 07:03:19.840512 12355 tablet.cc:2369] T 8f85cbc46f664234abf5b688100ab783 P 146299b0fdf74f4a9b52ab72aece32c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:19.855077 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.840650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.11.81.193:43405)
0620 07:03:19.840843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.840850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.841416 (+ 566us) server_negotiation.cc:407] Connection header received
0620 07:03:19.841551 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.841556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.841634 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.841726 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.843033 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.843861 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.845145 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.845603 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.845721 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.849575 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.849582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.849585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.849628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.852440 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.853165 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.853171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.853173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.853316 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.853708 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.853713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.853715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.854033 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.854270 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.854527 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.854722 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:03:19 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:19.898638 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.884385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.11.81.193:43405)
0620 07:03:19.884626 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.884633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.884806 (+ 173us) server_negotiation.cc:407] Connection header received
0620 07:03:19.885049 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.885054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.885144 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.885242 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.886748 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.887621 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.888937 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.889384 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.889513 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.893686 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.893692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.893695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.893733 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.896277 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.896889 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.896893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.896895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.897023 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.897418 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.897421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.897422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.897687 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.897872 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.898187 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.898370 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 07:03:19.900421 12269 tablet_service.cc:2908] Rejecting scan request for tablet 8f85cbc46f664234abf5b688100ab783: Uninitialized: clean time has not yet been initialized
I20260620 07:03:19.903368 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12226
I20260620 07:03:19.916328 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:19.904184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.11.81.193:43405)
0620 07:03:19.904338 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:19.904342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:19.904706 (+ 364us) server_negotiation.cc:407] Connection header received
0620 07:03:19.904839 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:19.904842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:19.904900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:19.904978 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:19.906152 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.906910 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.908291 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:19.908706 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:19.908798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:19.911064 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:19.911070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:19.911073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:19.911108 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:19.913767 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.914456 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.914463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.914467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.914665 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:19.915103 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:19.915107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:19.915109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:19.915434 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:19.915650 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:19.915879 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 07:03:19.916074 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:03:19.916951 12349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:20.008522 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12226
I20260620 07:03:20.023239 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12360
Jun 20 07:03:20 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:20.038547 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:20.024127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.11.81.194:45885)
0620 07:03:20.024299 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:20.024304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:20.024321 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:20.024538 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:20.024543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:20.024631 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:20.024716 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:20.026039 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:20.026873 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:20.028234 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:20.028647 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:20.028746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:20.032976 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:20.032992 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:20.033008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:20.033050 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:20.035919 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:20.036560 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:20.036567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:20.036569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:20.036743 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:20.037030 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:20.037033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:20.037035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:20.037400 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:20.037657 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:20.038005 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 07:03:20.038201 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:03:20.039258 12483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:20.126677 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12360
I20260620 07:03:20.140734 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12493
Jun 20 07:03:20 dist-test-slave-wvz3 krb5kdc[12120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:20.156199 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:20.141601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.11.81.195:44315)
0620 07:03:20.141822 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:20.141826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:20.141845 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:03:20.141911 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:20.141915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:20.141978 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:20.142096 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:20.143481 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:20.144260 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:20.145578 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:20.145974 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:20.146094 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:20.150320 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:20.150337 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:20.150355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:20.150393 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:20.153324 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:20.154080 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:20.154089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:20.154091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:20.154235 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:20.154611 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:20.154615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:20.154618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:20.154986 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:20.155318 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:20.155655 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:03:20.155869 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:03:20.156867 12616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:20.244155 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12493
I20260620 07:03:20.257104 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12133
I20260620 07:03:20.258373 12195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:20.365324 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12133
2026-06-20T07:03:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3992 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:20 dist-test-slave-wvz3 krb5kdc[12669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:20 dist-test-slave-wvz3 krb5kdc[12669](info): set up 2 sockets
Jun 20 07:03:20 dist-test-slave-wvz3 krb5kdc[12669](info): commencing operation
krb5kdc: starting...
W20260620 07:03:21.359051 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:21 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:21.538888 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36571
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:21.852010 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:21.852322 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:21.852423 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:21.859725 12682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:21.859838 12682 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:03:21.859884 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:21.859964 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:21.860030 12682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:21.860116 12682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:21.870301 12682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:36571
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:21.871631 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:21.872918 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:21.884909 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:21.884972 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:21.884940 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:21.885829 12682 server_base.cc:1034] running on GCE node
I20260620 07:03:21.886631 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:21.888144 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:21.889390 12682 hybrid_clock.cc:648] HybridClock initialized: now 1781939001889324 us; error 47 us; skew 500 ppm
I20260620 07:03:21.889766 12682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:21 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:21.892977 12682 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:21.894480 12682 webserver.cc:460] Webserver started at http://127.11.81.254:43377/ using document root <none> and password file <none>
I20260620 07:03:21.895226 12682 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:21.895355 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:21.895678 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:21.898631 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "9423765fdf43490ba3cb63dd5b5a13c2"
format_stamp: "Formatted at 2026-06-20 07:03:21 on dist-test-slave-wvz3"
server_key: "9cc0b1fbd4a6ec9d18263b7fd52e6b8d"
server_key_iv: "4947d9f127eea74757472cc6bbfc5e0d"
server_key_version: "encryptionkey@0"
I20260620 07:03:21.899456 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "9423765fdf43490ba3cb63dd5b5a13c2"
format_stamp: "Formatted at 2026-06-20 07:03:21 on dist-test-slave-wvz3"
server_key: "9cc0b1fbd4a6ec9d18263b7fd52e6b8d"
server_key_iv: "4947d9f127eea74757472cc6bbfc5e0d"
server_key_version: "encryptionkey@0"
I20260620 07:03:21.904873 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:03:21.908587 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:21.910164 12682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:21.910346 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "9423765fdf43490ba3cb63dd5b5a13c2"
format_stamp: "Formatted at 2026-06-20 07:03:21 on dist-test-slave-wvz3"
server_key: "9cc0b1fbd4a6ec9d18263b7fd52e6b8d"
server_key_iv: "4947d9f127eea74757472cc6bbfc5e0d"
server_key_version: "encryptionkey@0"
I20260620 07:03:21.910578 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:21.922012 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:21.926532 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:21.926803 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:21.947877 12682 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:36571
I20260620 07:03:21.947897 12758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:36571 every 8 connection(s)
I20260620 07:03:21.949307 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:21.953235 12759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:21.956493 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12682
I20260620 07:03:21.956696 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:21.957386 11591 external_mini_cluster.cc:1428] Setting key b6ea9bd1fe8cc6b7320c1155ff0441a7
I20260620 07:03:21.966140 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Bootstrap starting.
I20260620 07:03:21.970029 12759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:21.971443 12759 log.cc:824] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:03:21 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:21.974684 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: No bootstrap required, opened a new log
I20260620 07:03:21.981138 12762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:21.959592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.11.81.254:36571)
0620 07:03:21.960353 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:21.960366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:21.960408 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:03:21.962093 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:21.962118 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:21.962594 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:21.962958 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:21.964772 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:21.966121 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:21.967866 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:21.968638 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:21.968772 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:21.973733 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:21.973741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:21.973758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:21.973800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:21.976805 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:21.977664 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:21.977671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:21.977678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:21.977893 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:21.978273 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:21.978276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:21.978278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:21.978901 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:21.979137 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:21.979620 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 07:03:21.980063 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":342,"threads_started":1}
I20260620 07:03:21.983669 12759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } }
I20260620 07:03:21.984050 12759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:21.984172 12759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9423765fdf43490ba3cb63dd5b5a13c2, State: Initialized, Role: FOLLOWER
I20260620 07:03:21.984807 12759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } }
I20260620 07:03:21.985036 12759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:21.985158 12759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:21.985312 12759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:21.987447 12759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } }
I20260620 07:03:21.987937 12759 leader_election.cc:302] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9423765fdf43490ba3cb63dd5b5a13c2; no voters:
I20260620 07:03:21.988492 12759 leader_election.cc:288] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:21.988678 12764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:21.989722 12764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9423765fdf43490ba3cb63dd5b5a13c2, State: Running, Role: LEADER
I20260620 07:03:21.990244 12764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } }
I20260620 07:03:21.990696 12759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:21.993141 12766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9423765fdf43490ba3cb63dd5b5a13c2. Latest consensus state: current_term: 1 leader_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } } }
I20260620 07:03:21.993415 12766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:21.992986 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9423765fdf43490ba3cb63dd5b5a13c2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36571 } } }
I20260620 07:03:21.993563 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:21.994349 12773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:21.999951 12773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:22.010259 12773 catalog_manager.cc:1347] Generated new cluster ID: 436465c9fce04c12a9d7e322fbd64ee9
I20260620 07:03:22.010383 12773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:22.030501 12773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:22.032204 12773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:22.042310 12773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Generated new TSK 0
I20260620 07:03:22.043315 12773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:22.092396 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36571
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:22.429472 12785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:22.429778 12785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:22.429876 12785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:22.437179 12785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:22.437304 12785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:22.437410 12785 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:22.437498 12785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:22.447335 12785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:22.448601 12785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:22.449935 12785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:22.463495 12791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:22.463517 12790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:22.464030 12793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:22.464241 12785 server_base.cc:1034] running on GCE node
I20260620 07:03:22.464910 12785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:22.465955 12785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:22.467206 12785 hybrid_clock.cc:648] HybridClock initialized: now 1781939002467143 us; error 34 us; skew 500 ppm
I20260620 07:03:22.467559 12785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:22 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:22.471184 12785 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:22.472805 12785 webserver.cc:460] Webserver started at http://127.11.81.193:34549/ using document root <none> and password file <none>
I20260620 07:03:22.473555 12785 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:22.473647 12785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:22.473934 12785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:22.477063 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "4e6d49d074d34526822a2857bc6bd6df"
format_stamp: "Formatted at 2026-06-20 07:03:22 on dist-test-slave-wvz3"
server_key: "96a4e5e9ceb2dffd7a2cb5d85de928f6"
server_key_iv: "0f10493bc8e23692e679a5008bc1d2c0"
server_key_version: "encryptionkey@0"
I20260620 07:03:22.477875 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "4e6d49d074d34526822a2857bc6bd6df"
format_stamp: "Formatted at 2026-06-20 07:03:22 on dist-test-slave-wvz3"
server_key: "96a4e5e9ceb2dffd7a2cb5d85de928f6"
server_key_iv: "0f10493bc8e23692e679a5008bc1d2c0"
server_key_version: "encryptionkey@0"
I20260620 07:03:22.483683 12785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:22.487799 12800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:22.489398 12785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:22.489603 12785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "4e6d49d074d34526822a2857bc6bd6df"
format_stamp: "Formatted at 2026-06-20 07:03:22 on dist-test-slave-wvz3"
server_key: "96a4e5e9ceb2dffd7a2cb5d85de928f6"
server_key_iv: "0f10493bc8e23692e679a5008bc1d2c0"
server_key_version: "encryptionkey@0"
I20260620 07:03:22.489843 12785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:22.531514 12785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:22.536358 12785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:22.536652 12785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:22.539723 12785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:22.539884 12785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:03:22.540025 12785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:22.540091 12785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:03:22 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:22.566074 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:22.541348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:44255 (local address 127.11.81.254:36571)
0620 07:03:22.543902 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:22.543908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:22.543933 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:22.546030 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:22.546036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:22.546136 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:22.546243 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:22.548740 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.549891 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.552432 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.552995 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.553156 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:22.558267 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:22.558274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:22.558280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:22.558326 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:22.561804 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:22.563035 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:22.563042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:22.563045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:22.563211 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:22.563874 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:22.563879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:22.563883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:22.564175 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:22.564318 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:22.565388 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0620 07:03:22.565663 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2299,"thread_start_us":2199,"threads_started":1}
I20260620 07:03:22.566944 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:22.541307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.193:44255)
0620 07:03:22.543024 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:22.543113 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:22.545593 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:22.546595 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:22.546609 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:22.547523 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:22.548450 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:22.548503 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.550093 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.550099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:22.552237 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:22.552257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.553292 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.553302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:22.553636 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:22.554443 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:22.554477 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:22.557915 (+ 3438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:22.561993 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:22.562004 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:22.562028 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:22.562858 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:22.563351 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:22.563356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:22.563360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:22.563739 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:22.564318 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:22.564327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:22.564875 (+ 548us) client_negotiation.cc:769] Sending connection context
0620 07:03:22.565239 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 07:03:22.565690 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1073,"threads_started":1}
I20260620 07:03:22.587313 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:22.573052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:59847 (local address 127.11.81.254:36571)
0620 07:03:22.573278 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:22.573284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:22.574184 (+ 900us) server_negotiation.cc:407] Connection header received
0620 07:03:22.574365 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:22.574369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:22.574434 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:22.574537 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:22.575946 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.576814 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.579021 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.579661 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.579798 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:22.581678 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:22.581683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:22.581686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:22.581720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:22.584320 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:22.585169 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:22.585174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:22.585178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:22.585342 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:22.585806 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:22.585812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:22.585816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:22.586121 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:22.586257 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:22.586672 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 07:03:22.586905 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:03:22.587440 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:22.573099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.193:59847)
0620 07:03:22.574050 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:22.574077 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:22.574253 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:22.574593 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:22.574598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:22.575255 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:22.575719 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:22.575739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.576969 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.576975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:22.578782 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:22.578804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:22.579860 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:22.579868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:22.580041 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:22.580698 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:22.580714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:22.581473 (+ 759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:22.584456 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:22.584462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:22.584467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:22.585040 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:22.585491 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:22.585496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:22.585500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:22.585687 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:22.586241 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:22.586246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:22.586435 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 07:03:22.586564 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:03:22.586944 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":488,"threads_started":1}
I20260620 07:03:22.589947 12785 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:43425
I20260620 07:03:22.589958 12920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:43425 every 8 connection(s)
I20260620 07:03:22.591539 12785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:22.593449 12921 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36571
I20260620 07:03:22.593686 12921 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:22.594334 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12785
I20260620 07:03:22.594340 12921 heartbeater.cc:511] Master 127.11.81.254:36571 requested a full tablet report, sending...
I20260620 07:03:22.594515 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:22.595274 11591 external_mini_cluster.cc:1428] Setting key bc8ecfc3e498f5d750069ff277c302dc
I20260620 07:03:22.596582 12714 ts_manager.cc:194] Registered new tserver with Master: 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425)
I20260620 07:03:22.598094 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:44255
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:22.635834 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36571
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:22.954799 12925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:22.955085 12925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:22.955229 12925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:22.962808 12925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:22.962941 12925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:22.963074 12925 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:22.963173 12925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:22.973548 12925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:22.974862 12925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:22.976269 12925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:22.989769 12930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:22.989933 12933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:22.990038 12931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:22.990586 12925 server_base.cc:1034] running on GCE node
I20260620 07:03:22.991317 12925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:22.992328 12925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:22.993585 12925 hybrid_clock.cc:648] HybridClock initialized: now 1781939002993506 us; error 44 us; skew 500 ppm
I20260620 07:03:22.993937 12925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:22 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:23.002043 12925 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:23.003742 12925 webserver.cc:460] Webserver started at http://127.11.81.194:44797/ using document root <none> and password file <none>
I20260620 07:03:23.004492 12925 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:23.004601 12925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:23.004914 12925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:23.008008 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "3f36ce18f83f46479b84987f0aaff47c"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "abe08c983d899040b7fc7220cfbfce27"
server_key_iv: "56ff6a339ad2987c61b47166a0be48ce"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.008833 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "3f36ce18f83f46479b84987f0aaff47c"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "abe08c983d899040b7fc7220cfbfce27"
server_key_iv: "56ff6a339ad2987c61b47166a0be48ce"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.014341 12925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:03:23.018185 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:23.019704 12925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:23.019869 12925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "3f36ce18f83f46479b84987f0aaff47c"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "abe08c983d899040b7fc7220cfbfce27"
server_key_iv: "56ff6a339ad2987c61b47166a0be48ce"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.020103 12925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:23.036240 12925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:23.041083 12925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:23.041390 12925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:23.044480 12925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:23.044593 12925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:23.044714 12925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:23.044782 12925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:03:23 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939002, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:23.070093 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.045015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:39767 (local address 127.11.81.254:36571)
0620 07:03:23.045219 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.045226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.046930 (+ 1704us) server_negotiation.cc:407] Connection header received
0620 07:03:23.049015 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.049021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.049127 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.049236 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:23.051911 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.052883 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.056147 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.056602 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.056717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.062142 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:23.062150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:23.062155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:23.062200 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:23.065722 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.066862 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.066868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.066872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.067054 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.067574 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.067579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.067583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.067868 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:23.068041 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.069396 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.069729 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:03:23.071779 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.045814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.194:39767)
0620 07:03:23.046738 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.046780 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.048720 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.049556 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.049570 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.050565 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:23.051630 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.051666 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.053137 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.053144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.055900 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.055933 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.056868 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.056876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.057243 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.058290 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:23.058329 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:23.061839 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:23.065929 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.065937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.065953 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.066661 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.067174 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.067181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.067183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.067452 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.068019 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:23.068029 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:23.068690 (+ 661us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.069132 (+ 442us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.069838 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":386,"threads_started":1}
I20260620 07:03:23.092428 12925 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:37529
I20260620 07:03:23.092928 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.078299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.194:45337)
0620 07:03:23.078892 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.078907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.079073 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.079427 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.079431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.080169 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:23.080788 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.080801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.081978 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.081984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.083072 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.083084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.083943 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.083952 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.084164 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.084846 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:23.084871 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:23.085964 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:23.089445 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.089452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.089457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.090048 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.090634 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.090641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.090649 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.090841 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.091594 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:23.091600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:23.091830 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.091959 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.092362 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":326,"threads_started":1}
I20260620 07:03:23.093165 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.078320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:45337 (local address 127.11.81.254:36571)
0620 07:03:23.078495 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.078499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.079007 (+ 508us) server_negotiation.cc:407] Connection header received
0620 07:03:23.079203 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.079208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.079305 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.079398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:23.080948 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.081838 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.083222 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.083702 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.083846 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.086215 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:23.086220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:23.086224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:23.086260 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:23.089282 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.090250 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.090254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.090256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.090407 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.090968 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.090974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.090977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.091454 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:23.091593 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.092599 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.092805 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:03:23.092428 13059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:37529 every 8 connection(s)
I20260620 07:03:23.094139 12925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:23.096367 13060 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36571
I20260620 07:03:23.096593 13060 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:23.097363 13060 heartbeater.cc:511] Master 127.11.81.254:36571 requested a full tablet report, sending...
I20260620 07:03:23.098646 12714 ts_manager.cc:194] Registered new tserver with Master: 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529)
I20260620 07:03:23.099502 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:39767
I20260620 07:03:23.103863 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 12925
I20260620 07:03:23.104059 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:23.104478 11591 external_mini_cluster.cc:1428] Setting key 81caa6b217a3ba6a9dd6580ae595e40d
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:23.144035 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36571
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:23.470875 13064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:23.471175 13064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:23.471369 13064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:23.479720 13064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:23.479872 13064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:23.480013 13064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:23.480132 13064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:23.492058 13064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:23.493328 13064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:23.494828 13064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:23.507987 13069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:23.508041 13072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:23.508417 13070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:23.508865 13064 server_base.cc:1034] running on GCE node
I20260620 07:03:23.509482 13064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:23.510452 13064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:23.511659 13064 hybrid_clock.cc:648] HybridClock initialized: now 1781939003511578 us; error 52 us; skew 500 ppm
I20260620 07:03:23.511960 13064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:23 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:23.515038 13064 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:23.516551 13064 webserver.cc:460] Webserver started at http://127.11.81.195:39685/ using document root <none> and password file <none>
I20260620 07:03:23.517241 13064 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:23.517325 13064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:23.517637 13064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:23.520602 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "623477214a914fbd82a74720900e1061"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "430e05c176082838d706d9c2a200c459"
server_key_iv: "6956142248fbbb7212b61eacced4411b"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.521366 13064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "623477214a914fbd82a74720900e1061"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "430e05c176082838d706d9c2a200c459"
server_key_iv: "6956142248fbbb7212b61eacced4411b"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.526955 13064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:03:23.530642 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:23.532202 13064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:03:23.532366 13064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "623477214a914fbd82a74720900e1061"
format_stamp: "Formatted at 2026-06-20 07:03:23 on dist-test-slave-wvz3"
server_key: "430e05c176082838d706d9c2a200c459"
server_key_iv: "6956142248fbbb7212b61eacced4411b"
server_key_version: "encryptionkey@0"
I20260620 07:03:23.532584 13064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:23.565840 13064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:23.570127 13064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:23.570407 13064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:23.573287 13064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:23.573386 13064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:23.573519 13064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:23.573575 13064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:03:23 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:23.600600 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.574656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.195:52397)
0620 07:03:23.575567 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.575619 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.578234 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.579259 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.579275 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.580081 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:23.581152 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.581179 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.582769 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.582774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.584358 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.584371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.585364 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.585373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.585592 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.586346 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:23.586386 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:23.589934 (+ 3548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:23.594581 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.594594 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.594615 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.595438 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.595970 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.595976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.595980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.596276 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.596966 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:23.596975 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:23.597547 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.597927 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.598643 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260620 07:03:23.600914 12921 heartbeater.cc:503] Master 127.11.81.254:36571 was elected leader, sending a full tablet report...
I20260620 07:03:23.602090 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.574076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:52397 (local address 127.11.81.254:36571)
0620 07:03:23.574279 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.574285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.575768 (+ 1483us) server_negotiation.cc:407] Connection header received
0620 07:03:23.578700 (+ 2932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.578708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.578803 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.578909 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:23.581397 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.582579 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.584554 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.585090 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.585214 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.590307 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:23.590315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:23.590321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:23.590368 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:23.594365 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.595602 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.595608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.595611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.595842 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.596419 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.596424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.596427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.596814 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:23.596975 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.601402 (+ 4427us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.601720 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:03:23.620393 13064 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36341
I20260620 07:03:23.620411 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36341 every 8 connection(s)
I20260620 07:03:23.621456 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.607518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:46353 (local address 127.11.81.254:36571)
0620 07:03:23.607692 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.607697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.608521 (+ 824us) server_negotiation.cc:407] Connection header received
0620 07:03:23.608717 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.608724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.608825 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.608930 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:23.609817 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.610895 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.612392 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.613035 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.613166 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.615576 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:23.615581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:23.615584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:23.615615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:23.618634 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.619421 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.619427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.619431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.619593 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.620014 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.620017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.620019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.620282 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:23.620409 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.620877 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.621100 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:03:23.621548 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.607695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.11.81.195:46353)
0620 07:03:23.608382 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.608404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.608581 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.608978 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.608981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.609398 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:23.609687 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.609696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.611045 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.611054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.612253 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.612266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.613277 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.613283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.613447 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.614118 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:23.614146 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:23.615339 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:23.618755 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.618761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.618765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.619270 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.619736 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:23.619745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:23.619748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:23.619882 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:23.620404 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:23.620408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:23.620619 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.620747 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.621112 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":356,"threads_started":1}
I20260620 07:03:23.621773 13064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:23.623788 13199 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36571
I20260620 07:03:23.624639 13199 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:23.625382 13199 heartbeater.cc:511] Master 127.11.81.254:36571 requested a full tablet report, sending...
I20260620 07:03:23.626737 12714 ts_manager.cc:194] Registered new tserver with Master: 623477214a914fbd82a74720900e1061 (127.11.81.195:36341)
I20260620 07:03:23.627568 12714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:52397
I20260620 07:03:23.631961 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13064
I20260620 07:03:23.632120 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:23.632500 11591 external_mini_cluster.cc:1428] Setting key 69242feb5c220212fd2cf3e8882aee73
I20260620 07:03:23.635488 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:03:23 dist-test-slave-wvz3 krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:03:23 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:23.679675 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.666042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.11.81.254:36571)
0620 07:03:23.666239 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.666244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.666488 (+ 244us) server_negotiation.cc:407] Connection header received
0620 07:03:23.666619 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.666623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.666694 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.666791 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:23.668189 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.668973 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.670234 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.670768 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.670866 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.674555 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:23.674563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:23.674568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:23.674627 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:23.677149 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.677793 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.677797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.677799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.677925 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:23.678278 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:23.678283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:23.678286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:23.678571 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:23.678820 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.679089 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.679342 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 07:03:23.683992 12714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:23.690845 12714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:23.726796 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.704189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:43425 (local address 127.0.0.1:50030)
0620 07:03:23.718929 (+ 14740us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.718945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.719091 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.719643 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.719648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.719675 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:23.720012 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.720025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721613 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.721619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.724393 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.724408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.725736 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.725743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.725929 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.725992 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.726135 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.726347 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14619,"mutex_wait_us":193,"thread_start_us":8248,"threads_started":1}
I20260620 07:03:23.727334 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.704591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.11.81.195:36341)
0620 07:03:23.717774 (+ 13183us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.717784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.717821 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:03:23.717944 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.717953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.718254 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.718562 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:23.719715 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721491 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.724154 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.725705 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.725832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.726057 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.726563 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.726730 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13020,"thread_start_us":12842,"threads_started":1}
I20260620 07:03:23.727358 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.705039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.11.81.193:43425)
0620 07:03:23.718173 (+ 13134us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.718181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.719050 (+ 869us) server_negotiation.cc:407] Connection header received
0620 07:03:23.719250 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.719255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.719463 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.719641 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:23.720166 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721472 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.724568 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.725550 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.725675 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.725891 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.726909 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.727073 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12979,"thread_start_us":12855,"threads_started":1}
I20260620 07:03:23.728078 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.703695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:37529 (local address 127.0.0.1:50464)
0620 07:03:23.712941 (+ 9246us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.712993 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.713196 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.719188 (+ 5992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.719193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.719215 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:23.719679 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.719691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721650 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.721655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.724393 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.724408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.726076 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.726082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.726267 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.726297 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.727588 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.727722 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8906,"mutex_wait_us":99,"thread_start_us":13369,"threads_started":1}
I20260620 07:03:23.728096 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.704545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.11.81.194:37529)
0620 07:03:23.718580 (+ 14035us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.718588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.718615 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:03:23.718712 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.718720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.718963 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.719878 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:23.720093 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721490 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.724563 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.725886 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.726013 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.726201 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.727667 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.727843 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13873,"thread_start_us":13755,"threads_started":1}
I20260620 07:03:23.728605 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.703801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36341 (local address 127.0.0.1:40334)
0620 07:03:23.716819 (+ 13018us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.716840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.716976 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.718904 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.718916 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.718947 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:23.719444 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.719456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.721712 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.721717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.723970 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.723983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.726210 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.726217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.726372 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.726394 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.726463 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.726608 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12864,"mutex_wait_us":182,"thread_start_us":14878,"threads_started":1}
I20260620 07:03:23.731418 13129 tablet_service.cc:1467] Processing CreateTablet for tablet 2d830b1f2423499fa3756915f8319a4a (DEFAULT_TABLE table=test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:23.731386 12851 tablet_service.cc:1467] Processing CreateTablet for tablet 2d830b1f2423499fa3756915f8319a4a (DEFAULT_TABLE table=test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:23.731575 12990 tablet_service.cc:1467] Processing CreateTablet for tablet 2d830b1f2423499fa3756915f8319a4a (DEFAULT_TABLE table=test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:23.732642 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d830b1f2423499fa3756915f8319a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:23.732710 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d830b1f2423499fa3756915f8319a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:23.732704 12851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d830b1f2423499fa3756915f8319a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:23.743569 13218 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Bootstrap starting.
I20260620 07:03:23.746714 13219 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Bootstrap starting.
I20260620 07:03:23.747500 13218 tablet_bootstrap.cc:654] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:23.748579 13218 log.cc:824] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:23.749579 13220 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Bootstrap starting.
I20260620 07:03:23.750571 13219 tablet_bootstrap.cc:654] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:23.751763 13219 log.cc:824] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:23.753082 13220 tablet_bootstrap.cc:654] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:23.753141 13218 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: No bootstrap required, opened a new log
I20260620 07:03:23.753414 13218 ts_tablet_manager.cc:1397] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 07:03:23.754303 13220 log.cc:824] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:23.754503 13219 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: No bootstrap required, opened a new log
I20260620 07:03:23.754854 13219 ts_tablet_manager.cc:1397] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 07:03:23.757966 13220 tablet_bootstrap.cc:492] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: No bootstrap required, opened a new log
I20260620 07:03:23.758235 13220 ts_tablet_manager.cc:1397] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:03:23.762105 13219 raft_consensus.cc:348] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.762540 13219 raft_consensus.cc:374] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:23.762632 13219 raft_consensus.cc:729] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f36ce18f83f46479b84987f0aaff47c, State: Initialized, Role: FOLLOWER
I20260620 07:03:23.763268 13219 consensus_queue.cc:260] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.763638 13218 raft_consensus.cc:348] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.764068 13218 raft_consensus.cc:374] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:23.764168 13218 raft_consensus.cc:729] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623477214a914fbd82a74720900e1061, State: Initialized, Role: FOLLOWER
I20260620 07:03:23.764766 13218 consensus_queue.cc:260] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.766048 13199 heartbeater.cc:503] Master 127.11.81.254:36571 was elected leader, sending a full tablet report...
I20260620 07:03:23.766645 13218 ts_tablet_manager.cc:1428] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 07:03:23.768618 13220 raft_consensus.cc:348] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.769043 13220 raft_consensus.cc:374] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:23.769182 13220 raft_consensus.cc:729] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6d49d074d34526822a2857bc6bd6df, State: Initialized, Role: FOLLOWER
I20260620 07:03:23.769870 13220 consensus_queue.cc:260] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.770053 13060 heartbeater.cc:503] Master 127.11.81.254:36571 was elected leader, sending a full tablet report...
I20260620 07:03:23.771067 13219 ts_tablet_manager.cc:1428] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260620 07:03:23.772899 13220 ts_tablet_manager.cc:1428] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
W20260620 07:03:23.843971 12922 tablet.cc:2369] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:23.846485 13061 tablet.cc:2369] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:23.873644 13200 tablet.cc:2369] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:23.878121 13225 raft_consensus.cc:482] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:23.878414 13225 raft_consensus.cc:504] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.880134 13225 leader_election.cc:288] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529), 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425)
I20260620 07:03:23.887339 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.880405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:37529 (local address 127.11.81.195:51843)
0620 07:03:23.880722 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.880742 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.880864 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.881300 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.881305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.881335 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:23.881549 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.881558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.882882 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.882888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.885021 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.885036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.886528 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.886532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.886747 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.886785 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.886878 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.887012 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 07:03:23.887367 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.880621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:51843 (local address 127.11.81.194:37529)
0620 07:03:23.880820 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.880826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.880854 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:03:23.881058 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.881064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.881147 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.881266 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:23.881714 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.882718 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.885185 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.886363 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.886530 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.886601 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.886949 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.887083 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:03:23.887499 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.880650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:56955 (local address 127.11.81.193:43425)
0620 07:03:23.880833 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.880839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.881379 (+ 540us) server_negotiation.cc:407] Connection header received
0620 07:03:23.881585 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.881590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.881668 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.881769 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:23.882228 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.883130 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.885357 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.886557 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.886692 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.886739 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.887131 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.887266 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:03:23.887583 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.880436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:43425 (local address 127.11.81.195:56955)
0620 07:03:23.881265 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:23.881279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:23.881399 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:23.881789 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:23.881792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:23.881813 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:23.882076 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.882084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.883249 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.883254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.885215 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:23.885224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.886752 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.886758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:23.886927 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.886957 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:03:23.887040 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:03:23.887147 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":221,"thread_start_us":335,"threads_started":1}
I20260620 07:03:23.888754 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d830b1f2423499fa3756915f8319a4a" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c" is_pre_election: true
I20260620 07:03:23.888757 12871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d830b1f2423499fa3756915f8319a4a" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df" is_pre_election: true
I20260620 07:03:23.889220 13010 raft_consensus.cc:2454] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:23.889220 12871 raft_consensus.cc:2454] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:23.889845 13083 leader_election.cc:302] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f36ce18f83f46479b84987f0aaff47c, 4e6d49d074d34526822a2857bc6bd6df, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:23.890226 13225 raft_consensus.cc:2781] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:23.890357 13225 raft_consensus.cc:482] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:23.890445 13225 raft_consensus.cc:3037] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:23.892663 13225 raft_consensus.cc:504] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.893227 13225 leader_election.cc:288] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Requested vote from peers 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529), 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425)
I20260620 07:03:23.893761 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d830b1f2423499fa3756915f8319a4a" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c"
I20260620 07:03:23.893759 12871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d830b1f2423499fa3756915f8319a4a" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df"
I20260620 07:03:23.893941 13010 raft_consensus.cc:3037] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:23.893936 12871 raft_consensus.cc:3037] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:23.896546 13010 raft_consensus.cc:2454] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:23.896646 12871 raft_consensus.cc:2454] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:23.896991 13083 leader_election.cc:302] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f36ce18f83f46479b84987f0aaff47c, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:23.897246 13225 raft_consensus.cc:2781] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:23.897701 13225 raft_consensus.cc:686] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 1 LEADER]: Becoming Leader. State: Replica: 623477214a914fbd82a74720900e1061, State: Running, Role: LEADER
I20260620 07:03:23.898185 13225 consensus_queue.cc:237] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } }
I20260620 07:03:23.902397 12714 catalog_manager.cc:5495] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 623477214a914fbd82a74720900e1061 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20260620 07:03:23.961872 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:23.955881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.11.81.195:36341)
0620 07:03:23.956092 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:23.956098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:23.956177 (+ 79us) server_negotiation.cc:407] Connection header received
0620 07:03:23.956505 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:23.956509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:23.956599 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:23.956714 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:23.957076 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.957824 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.959726 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:23.960130 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:23.960226 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:23.960781 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:23.961301 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:23.961414 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:23.961554 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 07:03:23.961623 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:03:23.970058 12871 raft_consensus.cc:1261] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:23.970080 13010 raft_consensus.cc:1261] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:23.970958 13228 consensus_queue.cc:1035] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:23.971360 13225 consensus_queue.cc:1035] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:24.002154 12713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:24.002552 12713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:24.007054 12713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Sending DeleteTablet for 3 replicas of tablet 2d830b1f2423499fa3756915f8319a4a
I20260620 07:03:24.008222 13129 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d830b1f2423499fa3756915f8319a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:23 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260620 07:03:24.008303 12990 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d830b1f2423499fa3756915f8319a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:23 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50464
I20260620 07:03:24.009102 13241 tablet_replica.cc:330] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: stopping tablet replica
I20260620 07:03:24.009070 12851 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d830b1f2423499fa3756915f8319a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:23 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50030
I20260620 07:03:24.009675 13241 raft_consensus.cc:2229] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:24.010396 13241 raft_consensus.cc:2258] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.010879 12712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:24.011749 12712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:24.013108 13241 ts_tablet_manager.cc:1905] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.020350 13243 tablet_replica.cc:330] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: stopping tablet replica
I20260620 07:03:24.020649 12850 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0459334e334cedb2256e03f172382b (DEFAULT_TABLE table=test-table [id=469eec3eb48847408a19ef3856fb51ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:24.020856 12989 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0459334e334cedb2256e03f172382b (DEFAULT_TABLE table=test-table [id=469eec3eb48847408a19ef3856fb51ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:24.021162 13243 raft_consensus.cc:2229] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:24.021301 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0459334e334cedb2256e03f172382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.021427 12989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0459334e334cedb2256e03f172382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.021862 13243 raft_consensus.cc:2258] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.023677 13129 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0459334e334cedb2256e03f172382b (DEFAULT_TABLE table=test-table [id=469eec3eb48847408a19ef3856fb51ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:24.025920 13243 ts_tablet_manager.cc:1905] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.028635 13220 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Bootstrap starting.
I20260620 07:03:24.023850 13241 ts_tablet_manager.cc:1918] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:24.028981 13241 log.cc:1196] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/2d830b1f2423499fa3756915f8319a4a
I20260620 07:03:24.029655 13241 ts_tablet_manager.cc:1939] T 2d830b1f2423499fa3756915f8319a4a P 623477214a914fbd82a74720900e1061: Deleting consensus metadata
I20260620 07:03:24.030179 13220 tablet_bootstrap.cc:654] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.031855 12698 catalog_manager.cc:4841] TS 623477214a914fbd82a74720900e1061 (127.11.81.195:36341): tablet 2d830b1f2423499fa3756915f8319a4a (table test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]) successfully deleted
I20260620 07:03:24.024163 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0459334e334cedb2256e03f172382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.035795 13243 ts_tablet_manager.cc:1918] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:24.036316 13243 log.cc:1196] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/2d830b1f2423499fa3756915f8319a4a
I20260620 07:03:24.036911 13243 ts_tablet_manager.cc:1939] T 2d830b1f2423499fa3756915f8319a4a P 4e6d49d074d34526822a2857bc6bd6df: Deleting consensus metadata
I20260620 07:03:24.039206 13220 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: No bootstrap required, opened a new log
I20260620 07:03:24.039467 13220 ts_tablet_manager.cc:1397] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20260620 07:03:24.040241 13219 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Bootstrap starting.
I20260620 07:03:24.040450 13218 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Bootstrap starting.
I20260620 07:03:24.041103 13242 tablet_replica.cc:330] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: stopping tablet replica
I20260620 07:03:24.042001 13218 tablet_bootstrap.cc:654] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.042181 13219 tablet_bootstrap.cc:654] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.042423 13242 raft_consensus.cc:2229] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:24.043022 13242 raft_consensus.cc:2258] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.043140 13220 raft_consensus.cc:348] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.043475 13220 raft_consensus.cc:374] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.043608 13220 raft_consensus.cc:729] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6d49d074d34526822a2857bc6bd6df, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.043955 13220 consensus_queue.cc:260] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.044703 13220 ts_tablet_manager.cc:1428] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 07:03:24.048930 12699 catalog_manager.cc:4841] TS 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425): tablet 2d830b1f2423499fa3756915f8319a4a (table test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]) successfully deleted
I20260620 07:03:24.052975 13242 ts_tablet_manager.cc:1905] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.054239 13218 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: No bootstrap required, opened a new log
I20260620 07:03:24.054433 13218 ts_tablet_manager.cc:1397] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260620 07:03:24.055445 13218 raft_consensus.cc:348] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.055716 13218 raft_consensus.cc:374] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.055830 13218 raft_consensus.cc:729] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623477214a914fbd82a74720900e1061, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.056140 13218 consensus_queue.cc:260] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.056893 13218 ts_tablet_manager.cc:1428] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:03:24.058983 13219 tablet_bootstrap.cc:492] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: No bootstrap required, opened a new log
I20260620 07:03:24.059165 13219 ts_tablet_manager.cc:1397] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20260620 07:03:24.060129 13219 raft_consensus.cc:348] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.060385 13219 raft_consensus.cc:374] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.060495 13219 raft_consensus.cc:729] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f36ce18f83f46479b84987f0aaff47c, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.060765 13219 consensus_queue.cc:260] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.061636 13242 ts_tablet_manager.cc:1918] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:24.061825 13242 log.cc:1196] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/2d830b1f2423499fa3756915f8319a4a
I20260620 07:03:24.062033 13219 ts_tablet_manager.cc:1428] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:24.062476 13242 ts_tablet_manager.cc:1939] T 2d830b1f2423499fa3756915f8319a4a P 3f36ce18f83f46479b84987f0aaff47c: Deleting consensus metadata
I20260620 07:03:24.065033 12701 catalog_manager.cc:4841] TS 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529): tablet 2d830b1f2423499fa3756915f8319a4a (table test-table [id=639fdc69a3384e278a8b9f1ba2e4228b]) successfully deleted
I20260620 07:03:24.109771 13225 raft_consensus.cc:482] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:24.109989 13225 raft_consensus.cc:504] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.110726 13225 leader_election.cc:288] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425), 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529)
I20260620 07:03:24.111229 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0459334e334cedb2256e03f172382b" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c" is_pre_election: true
I20260620 07:03:24.111311 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0459334e334cedb2256e03f172382b" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df" is_pre_election: true
I20260620 07:03:24.111486 13010 raft_consensus.cc:2454] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:24.111642 12870 raft_consensus.cc:2454] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:24.111940 13083 leader_election.cc:302] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f36ce18f83f46479b84987f0aaff47c, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:24.112229 13225 raft_consensus.cc:2781] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:24.112367 13225 raft_consensus.cc:482] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:24.112462 13225 raft_consensus.cc:3037] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.114686 13225 raft_consensus.cc:504] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.115267 13225 leader_election.cc:288] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Requested vote from peers 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425), 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529)
I20260620 07:03:24.115727 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0459334e334cedb2256e03f172382b" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df"
I20260620 07:03:24.115906 12870 raft_consensus.cc:3037] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.115859 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0459334e334cedb2256e03f172382b" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c"
I20260620 07:03:24.116078 13010 raft_consensus.cc:3037] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.117918 12870 raft_consensus.cc:2454] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:24.118295 13081 leader_election.cc:302] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6d49d074d34526822a2857bc6bd6df, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:24.118578 13225 raft_consensus.cc:2781] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:24.118767 13225 raft_consensus.cc:686] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 1 LEADER]: Becoming Leader. State: Replica: 623477214a914fbd82a74720900e1061, State: Running, Role: LEADER
I20260620 07:03:24.119026 13010 raft_consensus.cc:2454] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:24.119081 13225 consensus_queue.cc:237] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.122056 12713 catalog_manager.cc:5495] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 623477214a914fbd82a74720900e1061 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } health_report { overall_health: HEALTHY } } }
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939001, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:24.181187 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.165970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36571 (local address 127.0.0.1:51994)
0620 07:03:24.166629 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:24.166650 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:24.166786 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:24.167175 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:24.167178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:24.167970 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:24.168237 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.168247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.169732 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.169736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.170758 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.170771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.171728 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.171734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.171856 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.172500 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:24.172529 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:24.174759 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:24.178576 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:24.178590 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:24.178596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:24.179050 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:24.179482 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:24.179487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:24.179490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:24.179641 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:24.180269 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:24.180275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:24.180451 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 07:03:24.180597 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:03:24.180823 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":354,"threads_started":1}
I20260620 07:03:24.181401 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.165971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.11.81.254:36571)
0620 07:03:24.166190 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.166197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.166742 (+ 545us) server_negotiation.cc:407] Connection header received
0620 07:03:24.166897 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.166903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.166998 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.167109 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.168390 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.169606 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.170902 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.171527 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.171653 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.174961 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.174968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.174973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.175011 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.178361 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.179170 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.179176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.179180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.179363 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.179742 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.179747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.179750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.180113 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.180256 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.180705 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.180986 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":67}
I20260620 07:03:24.187109 12713 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:03:24.188499 12713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:24.197343 12850 tablet_service.cc:1467] Processing CreateTablet for tablet 45cc4d8c570d46a59691968922b1d659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2cadaec038a4ce2a9f7f05dc33dd65d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:24.197444 12990 tablet_service.cc:1467] Processing CreateTablet for tablet 45cc4d8c570d46a59691968922b1d659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2cadaec038a4ce2a9f7f05dc33dd65d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:24.197932 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cc4d8c570d46a59691968922b1d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.197952 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cc4d8c570d46a59691968922b1d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.197835 13129 tablet_service.cc:1467] Processing CreateTablet for tablet 45cc4d8c570d46a59691968922b1d659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2cadaec038a4ce2a9f7f05dc33dd65d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:24.198375 13129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cc4d8c570d46a59691968922b1d659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:24.205207 13219 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: Bootstrap starting.
I20260620 07:03:24.205272 13220 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: Bootstrap starting.
I20260620 07:03:24.206631 13219 tablet_bootstrap.cc:654] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.206943 13220 tablet_bootstrap.cc:654] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.208636 13219 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: No bootstrap required, opened a new log
I20260620 07:03:24.208798 13219 ts_tablet_manager.cc:1397] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:03:24.209398 13219 raft_consensus.cc:348] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.209594 13219 raft_consensus.cc:374] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.209666 13219 raft_consensus.cc:729] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f36ce18f83f46479b84987f0aaff47c, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.209919 13219 consensus_queue.cc:260] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.210459 13224 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.210852 13224 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.211530 13219 ts_tablet_manager.cc:1428] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:24.212568 13218 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: Bootstrap starting.
I20260620 07:03:24.214366 13218 tablet_bootstrap.cc:654] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:24.216912 13218 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: No bootstrap required, opened a new log
I20260620 07:03:24.217087 13218 ts_tablet_manager.cc:1397] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260620 07:03:24.217705 13218 raft_consensus.cc:348] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.217919 13218 raft_consensus.cc:374] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.218012 13218 raft_consensus.cc:729] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623477214a914fbd82a74720900e1061, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.218252 13218 consensus_queue.cc:260] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.218638 13225 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.218977 13225 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.219507 13218 ts_tablet_manager.cc:1428] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:03:24.221560 13220 tablet_bootstrap.cc:492] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: No bootstrap required, opened a new log
I20260620 07:03:24.221738 13220 ts_tablet_manager.cc:1397] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20260620 07:03:24.222506 13220 raft_consensus.cc:348] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.222764 13220 raft_consensus.cc:374] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:24.222837 13220 raft_consensus.cc:729] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6d49d074d34526822a2857bc6bd6df, State: Initialized, Role: FOLLOWER
I20260620 07:03:24.223070 13220 consensus_queue.cc:260] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.223838 13220 ts_tablet_manager.cc:1428] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 07:03:24.223592 13226 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.224052 13226 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.297945 13225 raft_consensus.cc:482] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:24.298177 13225 raft_consensus.cc:504] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.298870 13225 leader_election.cc:288] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425), 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529)
I20260620 07:03:24.299501 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cc4d8c570d46a59691968922b1d659" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df" is_pre_election: true
I20260620 07:03:24.299494 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cc4d8c570d46a59691968922b1d659" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c" is_pre_election: true
I20260620 07:03:24.299768 13010 raft_consensus.cc:2454] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:24.299769 12870 raft_consensus.cc:2454] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 0.
I20260620 07:03:24.300201 13083 leader_election.cc:302] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f36ce18f83f46479b84987f0aaff47c, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:24.300519 13225 raft_consensus.cc:2781] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:24.300633 13225 raft_consensus.cc:482] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:24.300760 13225 raft_consensus.cc:3037] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.302878 13225 raft_consensus.cc:504] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.303704 13225 leader_election.cc:288] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Requested vote from peers 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425), 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529)
I20260620 07:03:24.303915 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cc4d8c570d46a59691968922b1d659" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6d49d074d34526822a2857bc6bd6df"
I20260620 07:03:24.304198 12870 raft_consensus.cc:3037] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.304165 13010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cc4d8c570d46a59691968922b1d659" candidate_uuid: "623477214a914fbd82a74720900e1061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f36ce18f83f46479b84987f0aaff47c"
I20260620 07:03:24.304365 13010 raft_consensus.cc:3037] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:24.306331 13010 raft_consensus.cc:2454] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:24.306730 13083 leader_election.cc:302] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f36ce18f83f46479b84987f0aaff47c, 623477214a914fbd82a74720900e1061; no voters:
I20260620 07:03:24.306999 13225 raft_consensus.cc:2781] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:24.307163 12870 raft_consensus.cc:2454] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623477214a914fbd82a74720900e1061 in term 1.
I20260620 07:03:24.307186 13225 raft_consensus.cc:686] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [term 1 LEADER]: Becoming Leader. State: Replica: 623477214a914fbd82a74720900e1061, State: Running, Role: LEADER
I20260620 07:03:24.307526 13225 consensus_queue.cc:237] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } }
I20260620 07:03:24.309177 13228 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: TxnStatusTablet state changed. Reason: New leader 623477214a914fbd82a74720900e1061. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.309530 13228 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:24.310412 13258 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:03:24.310675 13258 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:03:24.310832 12713 catalog_manager.cc:5495] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 623477214a914fbd82a74720900e1061 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } health_report { overall_health: HEALTHY } } }
I20260620 07:03:24.350533 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.344678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36341 (local address 127.0.0.1:40352)
0620 07:03:24.344922 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:24.344940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:24.345070 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:24.345590 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:24.345594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:24.345604 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:24.345771 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.345778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.347219 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.347223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.348406 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.348420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.349385 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.349397 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.349581 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.349644 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:24.350062 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:24.350074 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:03:24.350129 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 07:03:24.350244 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:03:24.350572 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.344818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.11.81.195:36341)
0620 07:03:24.345087 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.345093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.345125 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:03:24.345275 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.345281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.345405 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.345514 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:24.345907 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.347064 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.348552 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.349209 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.349374 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.349826 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:24.349970 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:24.350067 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.350183 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.350272 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:03:24.354630 12870 raft_consensus.cc:1261] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:24.354712 13010 raft_consensus.cc:1261] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:24.355389 13225 consensus_queue.cc:1035] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:24.355672 13228 consensus_queue.cc:1035] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:24.359870 13224 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: TxnStatusTablet state changed. Reason: New leader 623477214a914fbd82a74720900e1061. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.360203 13224 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.361536 13238 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.361963 13238 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:24.362325 13233 mvcc.cc:205] Tried to move back new op lower bound from 7298822161830191104 to 7298822161646026752. Current Snapshot: MvccSnapshot[applied={T|T < 7298822161830191104}]
I20260620 07:03:24.362589 13231 mvcc.cc:205] Tried to move back new op lower bound from 7298822161830191104 to 7298822161646026752. Current Snapshot: MvccSnapshot[applied={T|T < 7298822161830191104 or (T in {7298822161830191104})}]
I20260620 07:03:24.362772 13224 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.363005 13224 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 3f36ce18f83f46479b84987f0aaff47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.372027 13226 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: TxnStatusTablet state changed. Reason: New leader 623477214a914fbd82a74720900e1061. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.372370 13226 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.375566 13228 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.375829 13228 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 623477214a914fbd82a74720900e1061: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:24.377297 13226 tablet_replica.cc:439] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "623477214a914fbd82a74720900e1061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 } } peers { permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 } } peers { permanent_uuid: "623477214a914fbd82a74720900e1061" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36341 } } }
I20260620 07:03:24.377550 13226 tablet_replica.cc:442] T 45cc4d8c570d46a59691968922b1d659 P 4e6d49d074d34526822a2857bc6bd6df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:24.387239 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.381061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:50903 (local address 127.11.81.195:36341)
0620 07:03:24.381253 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.381259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.381745 (+ 486us) server_negotiation.cc:407] Connection header received
0620 07:03:24.381898 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.381904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.381988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.382079 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:24.382678 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.383846 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.385342 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.385997 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.386136 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.386565 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:24.386682 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:24.386751 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.386877 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.386944 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:03:24.387385 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.380972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36341 (local address 127.11.81.195:50903)
0620 07:03:24.381646 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:24.381662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:24.381781 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:24.382111 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:24.382115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:24.382124 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:24.382442 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.382450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.384010 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.384014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.385205 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:24.385217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.386174 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.386178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:24.386292 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.386351 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:24.386769 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:24.386779 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:03:24.386853 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:03:24.386990 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":353,"threads_started":1}
I20260620 07:03:24.398058 13010 raft_consensus.cc:1261] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:24.398058 12870 raft_consensus.cc:1261] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Refusing update from remote peer 623477214a914fbd82a74720900e1061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:24.398644 13225 consensus_queue.cc:1035] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6d49d074d34526822a2857bc6bd6df" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:24.398938 13228 consensus_queue.cc:1035] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f36ce18f83f46479b84987f0aaff47c" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:24.403431 13232 mvcc.cc:205] Tried to move back new op lower bound from 7298822162009300992 to 7298822160875159552. Current Snapshot: MvccSnapshot[applied={T|T < 7298822162009300992}]
I20260620 07:03:24.458843 12713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:24.459074 12713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:24.462293 12713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9423765fdf43490ba3cb63dd5b5a13c2: Sending DeleteTablet for 3 replicas of tablet 9e0459334e334cedb2256e03f172382b
I20260620 07:03:24.463096 12850 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0459334e334cedb2256e03f172382b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:24 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50030
I20260620 07:03:24.463160 12990 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0459334e334cedb2256e03f172382b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:24 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50464
I20260620 07:03:24.463384 13129 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0459334e334cedb2256e03f172382b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:24 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:40334
I20260620 07:03:24.463495 13243 tablet_replica.cc:330] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: stopping tablet replica
I20260620 07:03:24.463492 13242 tablet_replica.cc:330] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: stopping tablet replica
I20260620 07:03:24.463699 13242 raft_consensus.cc:2229] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:24.463697 13243 raft_consensus.cc:2229] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:24.463878 13243 raft_consensus.cc:2258] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.463891 13242 raft_consensus.cc:2258] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.464154 13241 tablet_replica.cc:330] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: stopping tablet replica
I20260620 07:03:24.464354 13241 raft_consensus.cc:2229] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:24.464696 13241 raft_consensus.cc:2258] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:24.465327 13243 ts_tablet_manager.cc:1905] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.466331 13241 ts_tablet_manager.cc:1905] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.470896 13243 ts_tablet_manager.cc:1918] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:24.471058 13243 log.cc:1196] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/9e0459334e334cedb2256e03f172382b
I20260620 07:03:24.471450 13243 ts_tablet_manager.cc:1939] T 9e0459334e334cedb2256e03f172382b P 4e6d49d074d34526822a2857bc6bd6df: Deleting consensus metadata
I20260620 07:03:24.472396 12699 catalog_manager.cc:4841] TS 4e6d49d074d34526822a2857bc6bd6df (127.11.81.193:43425): tablet 9e0459334e334cedb2256e03f172382b (table test-table [id=469eec3eb48847408a19ef3856fb51ad]) successfully deleted
I20260620 07:03:24.473973 13241 ts_tablet_manager.cc:1918] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:24.474119 13241 log.cc:1196] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/9e0459334e334cedb2256e03f172382b
I20260620 07:03:24.474552 13241 ts_tablet_manager.cc:1939] T 9e0459334e334cedb2256e03f172382b P 623477214a914fbd82a74720900e1061: Deleting consensus metadata
I20260620 07:03:24.476059 12698 catalog_manager.cc:4841] TS 623477214a914fbd82a74720900e1061 (127.11.81.195:36341): tablet 9e0459334e334cedb2256e03f172382b (table test-table [id=469eec3eb48847408a19ef3856fb51ad]) successfully deleted
I20260620 07:03:24.476325 13242 ts_tablet_manager.cc:1905] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:24.482828 13242 ts_tablet_manager.cc:1918] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:24.483038 13242 log.cc:1196] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/9e0459334e334cedb2256e03f172382b
I20260620 07:03:24.483609 13242 ts_tablet_manager.cc:1939] T 9e0459334e334cedb2256e03f172382b P 3f36ce18f83f46479b84987f0aaff47c: Deleting consensus metadata
I20260620 07:03:24.484975 12701 catalog_manager.cc:4841] TS 3f36ce18f83f46479b84987f0aaff47c (127.11.81.194:37529): tablet 9e0459334e334cedb2256e03f172382b (table test-table [id=469eec3eb48847408a19ef3856fb51ad]) successfully deleted
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:24.503322 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.480309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.11.81.193:43425)
0620 07:03:24.487715 (+ 7406us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.487719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.487741 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:03:24.487798 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.487802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.487878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.488002 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.489623 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.490466 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.491993 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.492442 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.492581 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.497130 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.497148 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.497164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.497203 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.500259 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.501020 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.501028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.501030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.501174 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.501595 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.501599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.501601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.501996 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.502329 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.502660 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.502894 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7273,"thread_start_us":7147,"threads_started":1}
W20260620 07:03:24.504295 12916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50034
I20260620 07:03:24.520879 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.507117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.11.81.254:36571)
0620 07:03:24.507381 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.507387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.507512 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:03:24.507643 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.507648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.507730 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.507836 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.509096 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.510239 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.511513 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.512153 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.512285 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.514717 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.514724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.514730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.514772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.518183 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.518867 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.518874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.518878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.519022 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.519387 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.519392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.519396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.519764 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.520027 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.520290 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.520515 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 07:03:24.521484 12713 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 07:03:24.523346 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12785
I20260620 07:03:24.536464 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.524552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.11.81.193:43425)
0620 07:03:24.524756 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.524762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.524921 (+ 159us) server_negotiation.cc:407] Connection header received
0620 07:03:24.525108 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.525113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.525195 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.525299 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.526349 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.527447 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.528668 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.529291 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.529418 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.531631 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.531638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.531643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.531684 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.534058 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.534640 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.534644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.534647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.534764 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.535096 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.535100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.535102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.535405 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.535650 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.535908 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.536155 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:03:24.537099 12916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50046
W20260620 07:03:24.537652 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:24.537858 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12785
I20260620 07:03:24.552937 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12925
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:24.568699 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.553777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.11.81.194:37529)
0620 07:03:24.554324 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.554329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.554346 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:24.554401 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.554405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.554471 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.554588 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.555951 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.556867 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.558186 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.558626 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.558753 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.563019 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.563038 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.563054 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.563092 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.566012 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.566683 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.566691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.566694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.566821 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.567178 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.567183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.567185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.567593 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.567876 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.568140 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.568355 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":261,"threads_started":1}
W20260620 07:03:24.569294 13055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50478
W20260620 07:03:24.569916 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12925: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:24.570073 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12925
I20260620 07:03:24.589746 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13064
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[12669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:24.605801 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:24.590484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.11.81.195:36341)
0620 07:03:24.590720 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:24.590727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:24.590891 (+ 164us) server_negotiation.cc:407] Connection header received
0620 07:03:24.591092 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:24.591096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:24.591167 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:24.591254 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:24.592545 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.593299 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.594487 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:24.595277 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:24.595492 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:24.599852 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:24.599871 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:24.599888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:24.599923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:24.602883 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.603657 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.603667 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.603671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.603814 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:24.604179 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:24.604186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:24.604188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:24.604591 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:24.604901 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:24.605257 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 07:03:24.605485 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 07:03:24.606537 13194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40366
W20260620 07:03:24.607178 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:24.607333 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13064
I20260620 07:03:24.624577 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 12682
I20260620 07:03:24.625771 12754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:24.739078 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 12682
2026-06-20T07:03:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4373 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[13278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[13278](info): set up 2 sockets
Jun 20 07:03:24 dist-test-slave-wvz3 krb5kdc[13278](info): commencing operation
krb5kdc: starting...
W20260620 07:03:25.743229 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:25 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:25.919504 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41401
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:26.233338 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:26.233657 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:26.233745 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:26.241139 13291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:26.241272 13291 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:03:26.241312 13291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:26.241367 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:26.241417 13291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:26.241485 13291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:26.251672 13291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:41401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:26.252970 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:26.254393 13291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:26.266372 13296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:26.266372 13299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:26.266520 13297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:26.267457 13291 server_base.cc:1034] running on GCE node
I20260620 07:03:26.268255 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:26.269784 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:26.271097 13291 hybrid_clock.cc:648] HybridClock initialized: now 1781939006271024 us; error 54 us; skew 500 ppm
I20260620 07:03:26.271512 13291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:26 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:26.274839 13291 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:26.276424 13291 webserver.cc:460] Webserver started at http://127.11.81.254:33633/ using document root <none> and password file <none>
I20260620 07:03:26.277168 13291 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:26.277261 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:26.277575 13291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:26.280752 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "52905da934194211a32b23bd3d8cd0a6"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "b05337d5aea2da153913b7979e3e2cf6"
server_key_iv: "4d2f861e8276b061c363838f968f8262"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.281622 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "52905da934194211a32b23bd3d8cd0a6"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "b05337d5aea2da153913b7979e3e2cf6"
server_key_iv: "4d2f861e8276b061c363838f968f8262"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.287204 13291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:03:26.291009 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:26.292575 13291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:26.292763 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "52905da934194211a32b23bd3d8cd0a6"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "b05337d5aea2da153913b7979e3e2cf6"
server_key_iv: "4d2f861e8276b061c363838f968f8262"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.292984 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:26.317471 13291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:26.321933 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:26.322198 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:26.343116 13291 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:41401
I20260620 07:03:26.343186 13367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:41401 every 8 connection(s)
I20260620 07:03:26.345088 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:26.347131 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13291
I20260620 07:03:26.347335 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:26.347985 11591 external_mini_cluster.cc:1428] Setting key 9a791dff8488f03f13399dbdb41406dc
I20260620 07:03:26.350281 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:03:26 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:26.363819 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Bootstrap starting.
I20260620 07:03:26.367623 13368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:26.368760 13368 log.cc:824] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:26.372023 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: No bootstrap required, opened a new log
I20260620 07:03:26.372870 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:26.350369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.11.81.254:41401)
0620 07:03:26.351188 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:26.351204 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:26.351253 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:03:26.353372 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:26.353401 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:26.353965 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:26.354438 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:26.355818 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.357830 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.359185 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.360215 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.360413 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:26.364621 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:26.364631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:26.364650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:26.364694 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:26.368342 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.369201 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.369208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.369215 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.369471 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.369901 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.369906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.369909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.370492 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:26.370702 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:26.371232 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 07:03:26.371668 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":328,"threads_started":1}
I20260620 07:03:26.379372 13368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } }
I20260620 07:03:26.379724 13368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:26.379818 13368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52905da934194211a32b23bd3d8cd0a6, State: Initialized, Role: FOLLOWER
I20260620 07:03:26.380353 13368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } }
I20260620 07:03:26.380544 13368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:26.380635 13368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:26.380746 13368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:26.383356 13368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } }
I20260620 07:03:26.383839 13368 leader_election.cc:302] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52905da934194211a32b23bd3d8cd0a6; no voters:
I20260620 07:03:26.384367 13368 leader_election.cc:288] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:26.384507 13373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:26.385119 13373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [term 1 LEADER]: Becoming Leader. State: Replica: 52905da934194211a32b23bd3d8cd0a6, State: Running, Role: LEADER
I20260620 07:03:26.385669 13373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } }
I20260620 07:03:26.386322 13368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:26.388334 13374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52905da934194211a32b23bd3d8cd0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } } }
I20260620 07:03:26.388685 13374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:26.389178 13375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52905da934194211a32b23bd3d8cd0a6. Latest consensus state: current_term: 1 leader_uuid: "52905da934194211a32b23bd3d8cd0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52905da934194211a32b23bd3d8cd0a6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41401 } } }
I20260620 07:03:26.389462 13375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:26.389597 13378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:26.396149 13378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:26.406461 13378 catalog_manager.cc:1347] Generated new cluster ID: abfd20892dff45cbbe2d050f9a5b1755
I20260620 07:03:26.406605 13378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:26.419099 13378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:26.420444 13378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:26.427053 13378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Generated new TSK 0
I20260620 07:03:26.427907 13378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:26.487461 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41401
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:26.825081 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:26.825430 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:26.825575 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:26.833101 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:26.833243 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:26.833369 13395 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:03:26.833434 13395 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:26.833542 13395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:26.843755 13395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:26.845067 13395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:26.846450 13395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:26.859658 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:26.859612 13403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:26.859639 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:26.860558 13395 server_base.cc:1034] running on GCE node
I20260620 07:03:26.861167 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:26.862154 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:26.863379 13395 hybrid_clock.cc:648] HybridClock initialized: now 1781939006863314 us; error 54 us; skew 500 ppm
I20260620 07:03:26.863708 13395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:26 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:26.866928 13395 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:26.868605 13395 webserver.cc:460] Webserver started at http://127.11.81.193:38495/ using document root <none> and password file <none>
I20260620 07:03:26.869298 13395 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:26.869412 13395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:26.869715 13395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:26.872771 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "359f3d3238a24f75860f53dbed3a52de"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "df4f1e59dbe46bbd45e732c4a7729d2c"
server_key_iv: "47674d7257afa3bdc5b82de89697cfde"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.873539 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "359f3d3238a24f75860f53dbed3a52de"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "df4f1e59dbe46bbd45e732c4a7729d2c"
server_key_iv: "47674d7257afa3bdc5b82de89697cfde"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.879103 13395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:26.882565 13410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:26.883932 13395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:26.884099 13395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "359f3d3238a24f75860f53dbed3a52de"
format_stamp: "Formatted at 2026-06-20 07:03:26 on dist-test-slave-wvz3"
server_key: "df4f1e59dbe46bbd45e732c4a7729d2c"
server_key_iv: "47674d7257afa3bdc5b82de89697cfde"
server_key_version: "encryptionkey@0"
I20260620 07:03:26.884279 13395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:26.906587 13395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:26.910856 13395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:26.911136 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:26.913774 13395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:26.913897 13395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:26.914019 13395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:26.914099 13395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:03:26 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:26.937000 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:26.914537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:44081 (local address 127.11.81.254:41401)
0620 07:03:26.914951 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:26.914955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:26.916835 (+ 1880us) server_negotiation.cc:407] Connection header received
0620 07:03:26.918964 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:26.918971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:26.919061 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:26.919145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:26.921580 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.922535 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.924921 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.925500 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.925628 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:26.930406 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:26.930412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:26.930415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:26.930451 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:26.933403 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.934314 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.934321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.934324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.934517 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.935067 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.935072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.935074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.935484 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:26.935652 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:26.936312 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 07:03:26.936596 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":199,"threads_started":1}
I20260620 07:03:26.937779 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:26.915828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.193:44081)
0620 07:03:26.916651 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:26.916730 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:26.918625 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:26.919755 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:26.919763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:26.920568 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:26.921403 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:26.921430 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.922680 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.922684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:26.924787 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:26.924800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.925788 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.925796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:26.926103 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:26.926943 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:26.926986 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:26.930163 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:26.933527 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:26.933536 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:26.933559 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:26.934163 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:26.934639 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:26.934644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:26.934648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:26.934899 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:26.935598 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:26.935605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:26.935966 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 07:03:26.936187 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 07:03:26.936592 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
I20260620 07:03:26.954205 13395 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:33077
I20260620 07:03:26.954241 13530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:33077 every 8 connection(s)
I20260620 07:03:26.955749 13395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:26.957821 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13395
I20260620 07:03:26.957832 13531 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41401
I20260620 07:03:26.958019 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:26.958050 13531 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:26.958642 11591 external_mini_cluster.cc:1428] Setting key f5653473f1ce41976fcd18ee8d58b706
I20260620 07:03:26.958775 13531 heartbeater.cc:511] Master 127.11.81.254:41401 requested a full tablet report, sending...
I20260620 07:03:26.961234 13322 ts_manager.cc:194] Registered new tserver with Master: 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077)
I20260620 07:03:26.963253 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:44081
I20260620 07:03:26.968583 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:26.946799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:51395 (local address 127.11.81.254:41401)
0620 07:03:26.947003 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:26.947008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:26.955907 (+ 8899us) server_negotiation.cc:407] Connection header received
0620 07:03:26.956082 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:26.956087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:26.956179 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:26.956311 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:26.957628 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.958651 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.960371 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.960928 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.961053 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:26.963167 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:26.963177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:26.963182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:26.963218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:26.966133 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.966749 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.966755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.966759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.966882 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:26.967203 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:26.967209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:26.967212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:26.967515 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:26.967659 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:26.968028 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 07:03:26.968244 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:03:26.968659 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:26.946797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.193:51395)
0620 07:03:26.955775 (+ 8978us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:26.955796 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:26.955964 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:26.956362 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:26.956367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:26.957085 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:26.957468 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:26.957484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.958840 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.958845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:26.960243 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:26.960256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:26.961117 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:26.961124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:26.961294 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:26.961953 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:26.961976 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:26.962956 (+ 980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:26.966261 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:26.966267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:26.966271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:26.966643 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:26.966982 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:26.966987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:26.966991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:26.967110 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:26.967629 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:26.967634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:26.967803 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 07:03:26.967919 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:03:26.968224 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8818,"thread_start_us":8677,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:26.999363 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41401
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:27.312935 13535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:27.313232 13535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:27.313333 13535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:27.320708 13535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:27.320874 13535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:27.320964 13535 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:03:27.321014 13535 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:27.321082 13535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:27.330837 13535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:27.332147 13535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:27.333372 13535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:27.346501 13540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:27.346501 13541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:27.346742 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:27.347622 13535 server_base.cc:1034] running on GCE node
I20260620 07:03:27.348230 13535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:27.349198 13535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:27.350427 13535 hybrid_clock.cc:648] HybridClock initialized: now 1781939007350367 us; error 52 us; skew 500 ppm
I20260620 07:03:27.350783 13535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:27 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:27.353960 13535 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:27.355480 13535 webserver.cc:460] Webserver started at http://127.11.81.194:34943/ using document root <none> and password file <none>
I20260620 07:03:27.356135 13535 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:27.356251 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:27.356544 13535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:27.359462 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "84291adf717b4993a42ad9bf8e0b4438"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "866d913841b1db20fed219cb4979008e"
server_key_iv: "4f18c0eb9e1fb357f4554be582c37e83"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.360217 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "84291adf717b4993a42ad9bf8e0b4438"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "866d913841b1db20fed219cb4979008e"
server_key_iv: "4f18c0eb9e1fb357f4554be582c37e83"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.365334 13535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:27.368881 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:27.370365 13535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:27.370540 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "84291adf717b4993a42ad9bf8e0b4438"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "866d913841b1db20fed219cb4979008e"
server_key_iv: "4f18c0eb9e1fb357f4554be582c37e83"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.370760 13535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:27.387880 13535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:27.392108 13535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:27.392375 13535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:27.395124 13535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:27.395259 13535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:27.395403 13535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:27.395486 13535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:03:27 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:27.418329 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.395706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:53849 (local address 127.11.81.254:41401)
0620 07:03:27.395891 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:27.395897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:27.397576 (+ 1679us) server_negotiation.cc:407] Connection header received
0620 07:03:27.399824 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:27.399831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:27.399950 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:27.400068 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:27.402933 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.403894 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.406873 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.407603 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.407733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.411581 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:27.411586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:27.411589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:27.411623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:27.414507 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.415681 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.415689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.415692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.415849 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.416537 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.416543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.416546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.416882 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:27.417027 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:27.417757 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 07:03:27.417994 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:03:27.419440 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.396438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.194:53849)
0620 07:03:27.397370 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:27.397409 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:27.399504 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:27.400553 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:27.400581 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:27.401602 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:27.402689 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.402730 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.404087 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.404095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.406642 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.406659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.407837 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.407842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.408027 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.408745 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:27.408767 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:27.411343 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:27.414654 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.414664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.414687 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.415491 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.416017 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.416022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.416025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.416396 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.417010 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:27.417018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:27.417388 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 07:03:27.417635 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 07:03:27.418187 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":389,"threads_started":1}
I20260620 07:03:27.440006 13535 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36687
I20260620 07:03:27.440114 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36687 every 8 connection(s)
I20260620 07:03:27.441660 13535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:27.442448 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.426618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60351 (local address 127.11.81.254:41401)
0620 07:03:27.426826 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:27.426831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:27.427853 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 07:03:27.428022 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:27.428027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:27.428109 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:27.428209 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:27.429643 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.430343 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.432414 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.433054 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.433198 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.435867 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:27.435872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:27.435875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:27.435911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:27.439145 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.440240 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.440246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.440250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.440404 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.440833 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.440838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.440841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.441316 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:27.441466 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:27.441872 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 07:03:27.442117 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:03:27.442561 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.426627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.194:60351)
0620 07:03:27.427755 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:27.427774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:27.427907 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:27.428273 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:27.428279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:27.428990 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:27.429476 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.429493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.430488 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.430494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.432248 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.432269 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.433281 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.433291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.433516 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.434202 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:27.434225 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:27.435657 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:27.439374 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.439381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.439385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.440098 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.440558 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.440564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.440568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.440712 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.441429 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:27.441433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:27.441612 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 07:03:27.441741 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:03:27.442030 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":837,"threads_started":1}
I20260620 07:03:27.445228 13670 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41401
I20260620 07:03:27.445430 13670 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:27.446074 13670 heartbeater.cc:511] Master 127.11.81.254:41401 requested a full tablet report, sending...
I20260620 07:03:27.447376 13323 ts_manager.cc:194] Registered new tserver with Master: 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687)
I20260620 07:03:27.447569 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13535
I20260620 07:03:27.447737 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:27.448277 11591 external_mini_cluster.cc:1428] Setting key ac47bb126b9bf10ad4f833e163532aa4
I20260620 07:03:27.448576 13323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:53849
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:27.490983 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41401
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:27.817548 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:27.817865 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:27.818004 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:27.825413 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:27.825557 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:27.825690 13674 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:03:27.825755 13674 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:03:27.825861 13674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:27.835645 13674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:27.836925 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:27.838240 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:27.851069 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:27.851136 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:27.851114 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:27.851917 13674 server_base.cc:1034] running on GCE node
I20260620 07:03:27.852552 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:27.853492 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:27.854696 13674 hybrid_clock.cc:648] HybridClock initialized: now 1781939007854657 us; error 73 us; skew 500 ppm
I20260620 07:03:27.854995 13674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:27 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:27.857982 13674 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:27.859534 13674 webserver.cc:460] Webserver started at http://127.11.81.195:39871/ using document root <none> and password file <none>
I20260620 07:03:27.860215 13674 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:27.860330 13674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:27.860616 13674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:27.863507 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "701a22a151e03e95633ca9a18c5dfeb4"
server_key_iv: "4c3879d5b3f11b5a45adbb7a9f1ffb13"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.864377 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "701a22a151e03e95633ca9a18c5dfeb4"
server_key_iv: "4c3879d5b3f11b5a45adbb7a9f1ffb13"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.869602 13674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:03:27.873090 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:27.874679 13674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:27.874845 13674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
format_stamp: "Formatted at 2026-06-20 07:03:27 on dist-test-slave-wvz3"
server_key: "701a22a151e03e95633ca9a18c5dfeb4"
server_key_iv: "4c3879d5b3f11b5a45adbb7a9f1ffb13"
server_key_version: "encryptionkey@0"
I20260620 07:03:27.875068 13674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:27.899175 13674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:27.903244 13674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:27.903551 13674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:27.906353 13674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:27.906472 13674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:27.906590 13674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:27.906723 13674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 07:03:27 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:27.931473 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.907608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.195:57427)
0620 07:03:27.908419 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:27.908468 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:27.910259 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:27.911084 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:27.911099 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:27.912006 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:27.913013 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.913048 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.914348 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.914353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.916971 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.916987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.918021 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.918028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.918354 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.919178 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:27.919204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:27.922673 (+ 3469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:27.926230 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.926241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.926256 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.926942 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.927459 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.927476 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.927480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.927794 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.928460 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:27.928468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:27.928970 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 07:03:27.929273 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 07:03:27.929854 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":356,"threads_started":1}
I20260620 07:03:27.932235 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.906826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:57427 (local address 127.11.81.254:41401)
0620 07:03:27.907001 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:27.907037 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:27.908609 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 07:03:27.910558 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:27.910563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:27.910658 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:27.910772 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:27.913245 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.914153 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.917154 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.917700 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.917830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.923015 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:27.923022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:27.923027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:27.923072 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:27.926048 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.927141 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.927147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.927151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.927344 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.927911 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.927916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.927920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.928241 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:27.928389 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:27.931560 (+ 3171us) server_negotiation.cc:299] Negotiation successful
0620 07:03:27.931857 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:03:27.951961 13674 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:44855
I20260620 07:03:27.952036 13808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:44855 every 8 connection(s)
I20260620 07:03:27.953467 13674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:27.957882 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.944028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:58565 (local address 127.11.81.254:41401)
0620 07:03:27.944223 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:27.944230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:27.944715 (+ 485us) server_negotiation.cc:407] Connection header received
0620 07:03:27.944914 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:27.944918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:27.944982 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:27.945059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:27.945961 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.946796 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.948529 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.949112 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.949230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.951722 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:27.951729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:27.951734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:27.951778 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:27.954989 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.955904 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.955910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.955913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.956071 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:27.956421 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:27.956425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:27.956429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:27.956797 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:27.956941 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:27.957332 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 07:03:27.957563 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:03:27.957718 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.944024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.11.81.195:58565)
0620 07:03:27.944600 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:27.944619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:27.944769 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:27.945111 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:27.945115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:27.945539 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:27.945821 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.945830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.946925 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.946930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.948322 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:27.948337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.949492 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:27.949499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:27.949669 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:27.950423 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:27.950441 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:27.951523 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:27.955160 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.955166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.955171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.955742 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.956180 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:27.956186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:27.956189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:27.956310 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:27.956920 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:27.956923 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:27.957087 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 07:03:27.957192 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:03:27.957431 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":311,"threads_started":1}
I20260620 07:03:27.959744 13809 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41401
I20260620 07:03:27.959967 13809 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:27.960644 13809 heartbeater.cc:511] Master 127.11.81.254:41401 requested a full tablet report, sending...
I20260620 07:03:27.961167 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13674
I20260620 07:03:27.961335 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:27.961848 11591 external_mini_cluster.cc:1428] Setting key 5a30088b7bca14bf4916838ba677d49e
I20260620 07:03:27.961915 13323 ts_manager.cc:194] Registered new tserver with Master: f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:27.962859 13323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:57427
I20260620 07:03:27.964856 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:27.966517 13531 heartbeater.cc:503] Master 127.11.81.254:41401 was elected leader, sending a full tablet report...
Jun 20 07:03:27 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:03:28 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:28.014948 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:27.995877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.11.81.254:41401)
0620 07:03:27.996106 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:27.996111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:27.996378 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:03:27.996551 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:27.996556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:27.996670 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:27.996773 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:27.998316 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:27.999144 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.000423 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.000860 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.000967 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.009405 (+ 8438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:28.009411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:28.009414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:28.009458 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:28.012225 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:28.012941 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:28.012946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:28.012948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:28.013080 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:28.013482 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:28.013486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:28.013488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:28.013788 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:28.014022 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.014353 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.014580 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 07:03:28.019269 13323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:28.026131 13323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:28.049176 13823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.038719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36687 (local address 127.0.0.1:46164)
0620 07:03:28.040436 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.040449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.040540 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.041445 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.041453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.041481 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.042267 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.042280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.044124 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.044129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.046917 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.046938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.048156 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.048339 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048386 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.048505 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.048704 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":187,"thread_start_us":978,"threads_started":1}
I20260620 07:03:28.049401 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.040692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.11.81.193:33077)
0620 07:03:28.041846 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.041854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.041879 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:28.041967 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.041973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.042178 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.042334 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.042968 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.044423 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.046953 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.048357 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048498 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048679 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.048980 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.049138 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":909,"threads_started":1}
I20260620 07:03:28.049732 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.039073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.11.81.195:44855)
0620 07:03:28.040848 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.040855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.040881 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:28.041017 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.041025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.041261 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.041431 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.042884 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.044080 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.046651 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.048106 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048230 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048450 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.049286 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.049442 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1618,"thread_start_us":1447,"threads_started":1}
I20260620 07:03:28.049968 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.039045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.11.81.194:36687)
0620 07:03:28.040298 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.040306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.040646 (+ 340us) server_negotiation.cc:407] Connection header received
0620 07:03:28.040752 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.040760 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.041026 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.041215 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.042478 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.043988 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.047060 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.047984 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048105 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048309 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.049420 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.049672 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":1702,"threads_started":1}
I20260620 07:03:28.050140 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.038589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44855 (local address 127.0.0.1:45182)
0620 07:03:28.040051 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.040091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.040255 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.041726 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.041730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.041753 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.042683 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.042695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.044217 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.044226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.046458 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.046476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.048613 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.048827 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048852 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.048928 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.049065 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"mutex_wait_us":3,"thread_start_us":698,"threads_started":1}
I20260620 07:03:28.049757 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.040177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.0.0.1:38612)
0620 07:03:28.041266 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.041285 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.041397 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.042339 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.042343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.042369 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.042770 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.042780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.044555 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.044560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.046802 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.046816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.048570 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.048576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.048767 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.048835 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.048921 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.049039 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":67,"thread_start_us":862,"threads_started":1}
I20260620 07:03:28.053587 13600 tablet_service.cc:1467] Processing CreateTablet for tablet ed98af25843648aa9bc3f853948ab7cd (DEFAULT_TABLE table=test-table [id=01d474648df045ababfec917e59df80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.054891 13600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed98af25843648aa9bc3f853948ab7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.054525 13461 tablet_service.cc:1467] Processing CreateTablet for tablet ed98af25843648aa9bc3f853948ab7cd (DEFAULT_TABLE table=test-table [id=01d474648df045ababfec917e59df80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.054523 13739 tablet_service.cc:1467] Processing CreateTablet for tablet ed98af25843648aa9bc3f853948ab7cd (DEFAULT_TABLE table=test-table [id=01d474648df045ababfec917e59df80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.055961 13461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed98af25843648aa9bc3f853948ab7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.056268 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed98af25843648aa9bc3f853948ab7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.066017 13828 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Bootstrap starting.
I20260620 07:03:28.069748 13828 tablet_bootstrap.cc:654] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.070227 13830 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Bootstrap starting.
I20260620 07:03:28.071121 13828 log.cc:824] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:28.071880 13829 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Bootstrap starting.
I20260620 07:03:28.073700 13830 tablet_bootstrap.cc:654] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.074916 13830 log.cc:824] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:28.075055 13829 tablet_bootstrap.cc:654] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.075461 13828 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: No bootstrap required, opened a new log
I20260620 07:03:28.075758 13828 ts_tablet_manager.cc:1397] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:03:28.076234 13829 log.cc:824] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:28.078198 13830 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: No bootstrap required, opened a new log
I20260620 07:03:28.078455 13830 ts_tablet_manager.cc:1397] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:03:28.079417 13829 tablet_bootstrap.cc:492] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: No bootstrap required, opened a new log
I20260620 07:03:28.079664 13829 ts_tablet_manager.cc:1397] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 07:03:28.086215 13828 raft_consensus.cc:348] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.086736 13828 raft_consensus.cc:374] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.086894 13828 raft_consensus.cc:729] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.086784 13829 raft_consensus.cc:348] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.087231 13829 raft_consensus.cc:374] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.087378 13829 raft_consensus.cc:729] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.087623 13828 consensus_queue.cc:260] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.088006 13829 consensus_queue.cc:260] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.088869 13830 raft_consensus.cc:348] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.089397 13830 raft_consensus.cc:374] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.089529 13830 raft_consensus.cc:729] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.090194 13830 consensus_queue.cc:260] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.092142 13830 ts_tablet_manager.cc:1428] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 07:03:28.094843 13670 heartbeater.cc:503] Master 127.11.81.254:41401 was elected leader, sending a full tablet report...
I20260620 07:03:28.095880 13828 ts_tablet_manager.cc:1428] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260620 07:03:28.097031 13809 heartbeater.cc:503] Master 127.11.81.254:41401 was elected leader, sending a full tablet report...
I20260620 07:03:28.097957 13829 ts_tablet_manager.cc:1428] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260620 07:03:28.127025 13834 raft_consensus.cc:482] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:28.127374 13834 raft_consensus.cc:504] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.129114 13834 leader_election.cc:288] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855), 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077)
I20260620 07:03:28.138224 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.129619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:52661 (local address 127.11.81.193:33077)
0620 07:03:28.129836 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.129842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.130551 (+ 709us) server_negotiation.cc:407] Connection header received
0620 07:03:28.130700 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.130705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.130781 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.130880 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.131742 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.132811 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.135683 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.136927 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.137110 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.137185 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.137843 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.137992 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:03:28.138244 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.129644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:42961 (local address 127.11.81.195:44855)
0620 07:03:28.129826 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.129832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.130500 (+ 668us) server_negotiation.cc:407] Connection header received
0620 07:03:28.130702 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.130706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.130781 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.130879 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.131691 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.132796 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.135682 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.136953 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.137108 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.137223 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.137862 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.138000 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:03:28.138345 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.129391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.11.81.194:52661)
0620 07:03:28.130413 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.130427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.130576 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.130937 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.130942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.130989 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.131562 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.131571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.132971 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.132975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.135474 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.135493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.137395 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.137406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.137676 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.137707 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.137792 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.137945 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":371,"spinlock_wait_cycles":61696,"thread_start_us":299,"threads_started":1}
I20260620 07:03:28.138892 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.129391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44855 (local address 127.11.81.194:42961)
0620 07:03:28.130352 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.130372 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.130561 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.130937 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.130942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.130973 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.131523 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.131540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.132953 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.132966 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.135482 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.135493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.137464 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.137470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.137663 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.137703 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.138394 (+ 691us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.138536 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":117,"thread_start_us":595,"threads_started":1}
I20260620 07:03:28.139521 13481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed98af25843648aa9bc3f853948ab7cd" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de" is_pre_election: true
I20260620 07:03:28.139839 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed98af25843648aa9bc3f853948ab7cd" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" is_pre_election: true
I20260620 07:03:28.140024 13481 raft_consensus.cc:2454] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 0.
I20260620 07:03:28.140321 13759 raft_consensus.cc:2454] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 0.
I20260620 07:03:28.140754 13551 leader_election.cc:302] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, 84291adf717b4993a42ad9bf8e0b4438; no voters:
I20260620 07:03:28.141167 13834 raft_consensus.cc:2781] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:28.141333 13834 raft_consensus.cc:482] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:28.141433 13834 raft_consensus.cc:3037] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.143613 13834 raft_consensus.cc:504] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.144202 13834 leader_election.cc:288] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 election: Requested vote from peers f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855), 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077)
I20260620 07:03:28.144730 13481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed98af25843648aa9bc3f853948ab7cd" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de"
I20260620 07:03:28.144744 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed98af25843648aa9bc3f853948ab7cd" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
I20260620 07:03:28.144917 13481 raft_consensus.cc:3037] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.144958 13759 raft_consensus.cc:3037] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.147186 13759 raft_consensus.cc:2454] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 1.
I20260620 07:03:28.147187 13481 raft_consensus.cc:2454] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 1.
I20260620 07:03:28.147673 13552 leader_election.cc:302] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84291adf717b4993a42ad9bf8e0b4438, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:28.148003 13834 raft_consensus.cc:2781] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:28.148452 13834 raft_consensus.cc:686] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 1 LEADER]: Becoming Leader. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Running, Role: LEADER
I20260620 07:03:28.148991 13834 consensus_queue.cc:237] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } }
I20260620 07:03:28.153715 13321 catalog_manager.cc:5495] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "84291adf717b4993a42ad9bf8e0b4438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } health_report { overall_health: UNKNOWN } } }
W20260620 07:03:28.194646 13671 tablet.cc:2369] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:03:28 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939006, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
W20260620 07:03:28.206393 13810 tablet.cc:2369] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:28.208250 13532 tablet.cc:2369] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:28.212163 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.198184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.11.81.254:41401)
0620 07:03:28.198395 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.198400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.199076 (+ 676us) server_negotiation.cc:407] Connection header received
0620 07:03:28.199263 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.199268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.199376 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.199533 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:28.200685 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.201494 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.202812 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.203337 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.203490 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.206885 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:28.206890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:28.206894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:28.206923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:28.209535 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:28.210190 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:28.210196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:28.210198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:28.210306 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:28.210664 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:28.210669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:28.210673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:28.210941 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:28.211073 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.211555 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.211863 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":83}
I20260620 07:03:28.212186 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.198247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41401 (local address 127.0.0.1:43934)
0620 07:03:28.198970 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.198986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.199128 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.199561 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.199564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.200278 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:28.200537 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.200547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.201639 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.201643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.202668 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.202680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.203538 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.203543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.203663 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.204449 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:28.204466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:28.206703 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:28.209670 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:28.209685 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:28.209693 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:28.210080 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:28.210411 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:28.210416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:28.210420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:28.210569 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:28.211090 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:28.211099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:28.211255 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.211448 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.211737 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":438,"threads_started":1}
I20260620 07:03:28.218619 13321 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:43934:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:03:28.219795 13321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:28.226538 13600 tablet_service.cc:1467] Processing CreateTablet for tablet b547f4c692594d6bb4946819cb5461d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2b6d88fa9340edb30633c9467841d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:28.226502 13461 tablet_service.cc:1467] Processing CreateTablet for tablet b547f4c692594d6bb4946819cb5461d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2b6d88fa9340edb30633c9467841d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:28.226616 13739 tablet_service.cc:1467] Processing CreateTablet for tablet b547f4c692594d6bb4946819cb5461d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e2b6d88fa9340edb30633c9467841d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:03:28.227381 13600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547f4c692594d6bb4946819cb5461d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.227442 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547f4c692594d6bb4946819cb5461d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.227473 13461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547f4c692594d6bb4946819cb5461d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.232317 13828 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: Bootstrap starting.
I20260620 07:03:28.233912 13828 tablet_bootstrap.cc:654] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.234579 13830 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: Bootstrap starting.
I20260620 07:03:28.235004 13829 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: Bootstrap starting.
I20260620 07:03:28.236311 13830 tablet_bootstrap.cc:654] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.236375 13829 tablet_bootstrap.cc:654] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.238860 13829 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: No bootstrap required, opened a new log
I20260620 07:03:28.238911 13830 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: No bootstrap required, opened a new log
I20260620 07:03:28.239014 13829 ts_tablet_manager.cc:1397] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:03:28.239040 13830 ts_tablet_manager.cc:1397] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:03:28.239307 13828 tablet_bootstrap.cc:492] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: No bootstrap required, opened a new log
I20260620 07:03:28.239454 13828 ts_tablet_manager.cc:1397] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260620 07:03:28.239784 13829 raft_consensus.cc:348] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.239789 13830 raft_consensus.cc:348] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.240031 13830 raft_consensus.cc:374] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.240031 13829 raft_consensus.cc:374] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.240130 13829 raft_consensus.cc:729] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.240165 13830 raft_consensus.cc:729] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.240099 13828 raft_consensus.cc:348] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.240329 13828 raft_consensus.cc:374] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.240437 13828 raft_consensus.cc:729] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.240381 13829 consensus_queue.cc:260] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.240439 13830 consensus_queue.cc:260] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.240654 13828 consensus_queue.cc:260] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.240964 13829 ts_tablet_manager.cc:1428] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:03:28.241000 13836 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.241238 13828 ts_tablet_manager.cc:1428] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 07:03:28.241092 13834 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.241335 13836 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.241389 13834 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.241271 13835 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.241583 13835 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.242774 13830 ts_tablet_manager.cc:1428] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:03:28.287807 13836 raft_consensus.cc:482] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:28.288086 13836 raft_consensus.cc:504] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.289661 13836 leader_election.cc:288] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:28.296115 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.289980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:58645 (local address 127.11.81.194:36687)
0620 07:03:28.290174 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.290180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.290801 (+ 621us) server_negotiation.cc:407] Connection header received
0620 07:03:28.290992 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.290997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.291054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.291124 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.291821 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.292511 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.294403 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.295189 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.295369 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.295443 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.295724 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.295852 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:03:28.296130 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.290034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:37613 (local address 127.11.81.195:44855)
0620 07:03:28.290212 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.290217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.290786 (+ 569us) server_negotiation.cc:407] Connection header received
0620 07:03:28.290980 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.290985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.291043 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.291117 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:28.291777 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.292576 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.294384 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.295192 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.295352 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.295439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.295755 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.295882 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:03:28.296249 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.289898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36687 (local address 127.11.81.193:58645)
0620 07:03:28.290699 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.290713 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.290859 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.291151 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.291155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.291200 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.291668 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.291678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.292639 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.292645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.294260 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.294271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.295435 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.295442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.295582 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.295606 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.295687 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.295820 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":186,"spinlock_wait_cycles":47872,"thread_start_us":338,"threads_started":1}
I20260620 07:03:28.296655 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.289898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44855 (local address 127.11.81.193:37613)
0620 07:03:28.290675 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.290695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.290848 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.291147 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.291151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.291197 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:28.291624 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.291636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.292701 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.292706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.294243 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.294255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.295484 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.295491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.295621 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.295642 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.295718 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.295850 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":12,"spinlock_wait_cycles":38784,"thread_start_us":425,"threads_started":1}
I20260620 07:03:28.297230 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547f4c692594d6bb4946819cb5461d7" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" is_pre_election: true
I20260620 07:03:28.297101 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547f4c692594d6bb4946819cb5461d7" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438" is_pre_election: true
I20260620 07:03:28.297467 13759 raft_consensus.cc:2454] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 0.
I20260620 07:03:28.297516 13620 raft_consensus.cc:2454] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 0.
I20260620 07:03:28.297950 13412 leader_election.cc:302] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:28.298537 13836 raft_consensus.cc:2781] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:28.298631 13836 raft_consensus.cc:482] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:28.298697 13836 raft_consensus.cc:3037] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.301220 13836 raft_consensus.cc:504] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.301777 13836 leader_election.cc:288] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 election: Requested vote from peers 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:28.302273 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547f4c692594d6bb4946819cb5461d7" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438"
I20260620 07:03:28.302281 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547f4c692594d6bb4946819cb5461d7" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
I20260620 07:03:28.302500 13759 raft_consensus.cc:3037] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.302538 13620 raft_consensus.cc:3037] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.305394 13759 raft_consensus.cc:2454] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 1.
I20260620 07:03:28.305408 13620 raft_consensus.cc:2454] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 1.
I20260620 07:03:28.305830 13412 leader_election.cc:302] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:28.306125 13836 raft_consensus.cc:2781] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:28.306574 13836 raft_consensus.cc:686] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [term 1 LEADER]: Becoming Leader. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Running, Role: LEADER
I20260620 07:03:28.307093 13836 consensus_queue.cc:237] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.310225 13852 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: TxnStatusTablet state changed. Reason: New leader 359f3d3238a24f75860f53dbed3a52de. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.310647 13852 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:28.311398 13854 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:03:28.311689 13854 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:03:28.311620 13323 catalog_manager.cc:5495] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de reported cstate change: term changed from 0 to 1, leader changed from <none> to 359f3d3238a24f75860f53dbed3a52de (127.11.81.193). New cstate: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } health_report { overall_health: UNKNOWN } } }
I20260620 07:03:28.330094 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.324137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.0.0.1:38618)
0620 07:03:28.324347 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.324363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.324513 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.324983 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.324988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.325000 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:28.325186 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.325194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.326251 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.326257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.327500 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.327517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.328487 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.328491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.328637 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.328694 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.329676 (+ 982us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.329691 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.329744 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.329855 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 07:03:28.330375 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.324080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.11.81.193:33077)
0620 07:03:28.324248 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.324255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.324501 (+ 246us) server_negotiation.cc:407] Connection header received
0620 07:03:28.324733 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.324737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.324812 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.324931 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:28.325307 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.326128 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.327657 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.328338 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.328484 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.328879 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.329578 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.329736 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.329785 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.329876 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:03:28.337873 13620 raft_consensus.cc:1261] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Refusing update from remote peer 359f3d3238a24f75860f53dbed3a52de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.337901 13759 raft_consensus.cc:1261] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Refusing update from remote peer 359f3d3238a24f75860f53dbed3a52de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.338828 13852 consensus_queue.cc:1035] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Connected to new peer: Peer: permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:28.339223 13836 consensus_queue.cc:1035] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:28.355880 13834 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: TxnStatusTablet state changed. Reason: New leader 359f3d3238a24f75860f53dbed3a52de. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.356249 13834 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.360586 13858 mvcc.cc:205] Tried to move back new op lower bound from 7298822178142420992 to 7298822178029465600. Current Snapshot: MvccSnapshot[applied={T|T < 7298822178142420992}]
I20260620 07:03:28.365738 13834 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.366029 13834 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 84291adf717b4993a42ad9bf8e0b4438: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.375110 13835 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: TxnStatusTablet state changed. Reason: New leader 359f3d3238a24f75860f53dbed3a52de. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.376127 13835 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.377311 13852 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.377599 13852 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:28.379055 13852 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.379323 13852 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P 359f3d3238a24f75860f53dbed3a52de: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:03:28.395438 13835 tablet_replica.cc:439] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } } }
I20260620 07:03:28.395709 13835 tablet_replica.cc:442] T b547f4c692594d6bb4946819cb5461d7 P f4de2f58f06f4bd3bfd9593c992e04ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:03:28.402633 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.396152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.11.81.194:36687)
0620 07:03:28.396326 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.396333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.396351 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:28.396452 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.396470 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.396569 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.396718 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:28.397557 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.398704 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.400131 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.400764 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.400886 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.401305 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.402017 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.402173 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.402219 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.402311 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:03:28.419790 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.413503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:38915 (local address 127.11.81.193:33077)
0620 07:03:28.413715 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.413722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.414255 (+ 533us) server_negotiation.cc:407] Connection header received
0620 07:03:28.414432 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.414438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.414517 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.414594 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:28.415157 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.415987 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.417957 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.418417 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.418517 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.419059 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.419192 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.419269 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.419483 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.419557 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:03:28.419943 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.413484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.11.81.194:38915)
0620 07:03:28.414142 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.414168 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.414313 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.414632 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.414637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.414651 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:28.414980 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.414991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.416100 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.416106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.417789 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.417805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.418590 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.418597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.418758 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.418830 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.419310 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.419321 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.419404 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.419558 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":307,"threads_started":1}
I20260620 07:03:28.430266 13758 raft_consensus.cc:1261] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Refusing update from remote peer 84291adf717b4993a42ad9bf8e0b4438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.430544 13481 raft_consensus.cc:1261] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Refusing update from remote peer 84291adf717b4993a42ad9bf8e0b4438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.430882 13839 consensus_queue.cc:1035] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:28.431248 13834 consensus_queue.cc:1035] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:28.439651 13857 mvcc.cc:205] Tried to move back new op lower bound from 7298822178524303360 to 7298822177382293504. Current Snapshot: MvccSnapshot[applied={T|T < 7298822178524303360}]
I20260620 07:03:28.443432 13860 mvcc.cc:205] Tried to move back new op lower bound from 7298822178524303360 to 7298822177382293504. Current Snapshot: MvccSnapshot[applied={T|T < 7298822178524303360}]
I20260620 07:03:28.473698 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.465315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:37159 (local address 127.11.81.194:36687)
0620 07:03:28.465506 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:28.465512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:28.468018 (+ 2506us) server_negotiation.cc:407] Connection header received
0620 07:03:28.468263 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:28.468269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:28.468338 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:28.468412 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:28.469066 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.470103 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.471896 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.472349 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.472453 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.472911 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.473062 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.473144 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:28.473332 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:03:28.473426 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:03:28.474762 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:28.465549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36687 (local address 127.11.81.193:37159)
0620 07:03:28.467918 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:28.467938 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:28.468133 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:28.468462 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:28.468467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:28.468479 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:28.468882 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.468897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.470285 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.470290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.471765 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:28.471778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:28.472535 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:28.472542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:28.472681 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:28.472744 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:28.473194 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:28.473204 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:03:28.474202 (+ 998us) client_negotiation.cc:240] Negotiation successful
0620 07:03:28.474348 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"thread_start_us":1970,"threads_started":1}
I20260620 07:03:28.524960 13323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:28.525251 13323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:28.529393 13323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Sending DeleteTablet for 3 replicas of tablet ed98af25843648aa9bc3f853948ab7cd
I20260620 07:03:28.531118 13739 tablet_service.cc:1514] Processing DeleteTablet for tablet ed98af25843648aa9bc3f853948ab7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:28 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 07:03:28.531819 13323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:03:28.532043 13461 tablet_service.cc:1514] Processing DeleteTablet for tablet ed98af25843648aa9bc3f853948ab7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:28 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38612
W20260620 07:03:28.532475 13323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:28.533180 13600 tablet_service.cc:1514] Processing DeleteTablet for tablet ed98af25843648aa9bc3f853948ab7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:28 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46164
I20260620 07:03:28.548060 13460 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc (DEFAULT_TABLE table=test-table [id=57bac543d158419f9079e139e56e3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.548308 13738 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc (DEFAULT_TABLE table=test-table [id=57bac543d158419f9079e139e56e3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.548691 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7a8c9d6a194f08a689451f793e1bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.548754 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7a8c9d6a194f08a689451f793e1bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.552099 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc (DEFAULT_TABLE table=test-table [id=57bac543d158419f9079e139e56e3189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:28.552688 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7a8c9d6a194f08a689451f793e1bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:28.556962 13829 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Bootstrap starting.
I20260620 07:03:28.558710 13829 tablet_bootstrap.cc:654] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.567631 13830 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Bootstrap starting.
I20260620 07:03:28.569392 13830 tablet_bootstrap.cc:654] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.570077 13878 tablet_replica.cc:330] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: stopping tablet replica
I20260620 07:03:28.570492 13878 raft_consensus.cc:2229] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:28.570884 13878 raft_consensus.cc:2258] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:28.571683 13877 tablet_replica.cc:330] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: stopping tablet replica
I20260620 07:03:28.572109 13877 raft_consensus.cc:2229] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:28.572427 13877 raft_consensus.cc:2258] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:28.573074 13830 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: No bootstrap required, opened a new log
I20260620 07:03:28.573258 13830 ts_tablet_manager.cc:1397] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 07:03:28.574041 13830 raft_consensus.cc:348] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.574308 13830 raft_consensus.cc:374] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.574332 13877 ts_tablet_manager.cc:1905] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:28.574381 13830 raft_consensus.cc:729] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.574683 13830 consensus_queue.cc:260] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.575443 13830 ts_tablet_manager.cc:1428] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 07:03:28.575722 13828 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Bootstrap starting.
I20260620 07:03:28.577309 13828 tablet_bootstrap.cc:654] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:28.582515 13877 ts_tablet_manager.cc:1918] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:28.582729 13877 log.cc:1196] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/ed98af25843648aa9bc3f853948ab7cd
I20260620 07:03:28.583252 13877 ts_tablet_manager.cc:1939] T ed98af25843648aa9bc3f853948ab7cd P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting consensus metadata
I20260620 07:03:28.584542 13829 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: No bootstrap required, opened a new log
I20260620 07:03:28.584730 13829 ts_tablet_manager.cc:1397] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.001s
I20260620 07:03:28.585216 13308 catalog_manager.cc:4841] TS f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855): tablet ed98af25843648aa9bc3f853948ab7cd (table test-table [id=01d474648df045ababfec917e59df80b]) successfully deleted
I20260620 07:03:28.584559 13879 tablet_replica.cc:330] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: stopping tablet replica
I20260620 07:03:28.585500 13829 raft_consensus.cc:348] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.585743 13829 raft_consensus.cc:374] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.585829 13829 raft_consensus.cc:729] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.586066 13829 consensus_queue.cc:260] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.586624 13829 ts_tablet_manager.cc:1428] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 07:03:28.586820 13879 raft_consensus.cc:2229] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:28.587654 13879 raft_consensus.cc:2258] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:28.589639 13878 ts_tablet_manager.cc:1905] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:28.593125 13879 ts_tablet_manager.cc:1905] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:28.588075 13828 tablet_bootstrap.cc:492] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: No bootstrap required, opened a new log
I20260620 07:03:28.595701 13828 ts_tablet_manager.cc:1397] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20260620 07:03:28.596799 13828 raft_consensus.cc:348] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.597059 13828 raft_consensus.cc:374] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:28.597134 13828 raft_consensus.cc:729] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Initialized, Role: FOLLOWER
I20260620 07:03:28.597402 13828 consensus_queue.cc:260] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.597781 13878 ts_tablet_manager.cc:1918] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:28.597962 13878 log.cc:1196] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/ed98af25843648aa9bc3f853948ab7cd
I20260620 07:03:28.597998 13828 ts_tablet_manager.cc:1428] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:03:28.598565 13878 ts_tablet_manager.cc:1939] T ed98af25843648aa9bc3f853948ab7cd P 359f3d3238a24f75860f53dbed3a52de: Deleting consensus metadata
I20260620 07:03:28.602259 13307 catalog_manager.cc:4841] TS 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077): tablet ed98af25843648aa9bc3f853948ab7cd (table test-table [id=01d474648df045ababfec917e59df80b]) successfully deleted
I20260620 07:03:28.602628 13879 ts_tablet_manager.cc:1918] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:28.602800 13879 log.cc:1196] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/ed98af25843648aa9bc3f853948ab7cd
I20260620 07:03:28.603353 13879 ts_tablet_manager.cc:1939] T ed98af25843648aa9bc3f853948ab7cd P 84291adf717b4993a42ad9bf8e0b4438: Deleting consensus metadata
I20260620 07:03:28.605285 13307 catalog_manager.cc:4841] TS 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687): tablet ed98af25843648aa9bc3f853948ab7cd (table test-table [id=01d474648df045ababfec917e59df80b]) successfully deleted
I20260620 07:03:28.626971 13834 raft_consensus.cc:482] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:28.627202 13834 raft_consensus.cc:504] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.628034 13834 leader_election.cc:288] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:28.628681 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7a8c9d6a194f08a689451f793e1bbc" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de" is_pre_election: true
I20260620 07:03:28.628814 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7a8c9d6a194f08a689451f793e1bbc" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" is_pre_election: true
I20260620 07:03:28.628969 13480 raft_consensus.cc:2454] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 0.
I20260620 07:03:28.629074 13759 raft_consensus.cc:2454] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 0.
I20260620 07:03:28.629473 13551 leader_election.cc:302] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, 84291adf717b4993a42ad9bf8e0b4438; no voters:
I20260620 07:03:28.629948 13834 raft_consensus.cc:2781] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:28.630075 13834 raft_consensus.cc:482] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:28.630167 13834 raft_consensus.cc:3037] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.633257 13834 raft_consensus.cc:504] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.634297 13834 leader_election.cc:288] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 election: Requested vote from peers 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:28.634418 13759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7a8c9d6a194f08a689451f793e1bbc" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
I20260620 07:03:28.634691 13759 raft_consensus.cc:3037] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.634692 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7a8c9d6a194f08a689451f793e1bbc" candidate_uuid: "84291adf717b4993a42ad9bf8e0b4438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de"
I20260620 07:03:28.634902 13480 raft_consensus.cc:3037] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:28.638154 13759 raft_consensus.cc:2454] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 1.
I20260620 07:03:28.638222 13480 raft_consensus.cc:2454] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84291adf717b4993a42ad9bf8e0b4438 in term 1.
I20260620 07:03:28.638595 13552 leader_election.cc:302] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84291adf717b4993a42ad9bf8e0b4438, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:28.639110 13834 raft_consensus.cc:2781] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:28.639326 13834 raft_consensus.cc:686] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 1 LEADER]: Becoming Leader. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Running, Role: LEADER
I20260620 07:03:28.639905 13834 consensus_queue.cc:237] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:28.642722 13323 catalog_manager.cc:5495] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "84291adf717b4993a42ad9bf8e0b4438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } health_report { overall_health: UNKNOWN } } }
I20260620 07:03:28.673859 13480 raft_consensus.cc:1261] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Refusing update from remote peer 84291adf717b4993a42ad9bf8e0b4438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.674156 13759 raft_consensus.cc:1261] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Refusing update from remote peer 84291adf717b4993a42ad9bf8e0b4438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:28.674762 13875 consensus_queue.cc:1035] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:28.675336 13875 consensus_queue.cc:1035] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:03:28.722204 13858 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:03:30.324162 13323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:30.324400 13323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:30.328387 13323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Sending DeleteTablet for 3 replicas of tablet 4f7a8c9d6a194f08a689451f793e1bbc
I20260620 07:03:30.329223 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38612
I20260620 07:03:30.329260 13600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46164
I20260620 07:03:30.329325 13739 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7a8c9d6a194f08a689451f793e1bbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 07:03:30.339170 13907 tablet_replica.cc:330] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: stopping tablet replica
I20260620 07:03:30.339682 13906 tablet_replica.cc:330] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: stopping tablet replica
I20260620 07:03:30.339922 13906 raft_consensus.cc:2229] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:30.340114 13908 tablet_replica.cc:330] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: stopping tablet replica
I20260620 07:03:30.340171 13907 raft_consensus.cc:2229] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:30.340210 13906 raft_consensus.cc:2258] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.340381 13908 raft_consensus.cc:2229] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:30.340677 13908 raft_consensus.cc:2258] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.340660 13907 raft_consensus.cc:2258] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.341557 13906 ts_tablet_manager.cc:1905] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.341646 13907 ts_tablet_manager.cc:1905] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.341681 13908 ts_tablet_manager.cc:1905] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.348304 13907 ts_tablet_manager.cc:1918] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:30.348335 13906 ts_tablet_manager.cc:1918] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:30.348479 13906 log.cc:1196] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/4f7a8c9d6a194f08a689451f793e1bbc
I20260620 07:03:30.348569 13907 log.cc:1196] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/4f7a8c9d6a194f08a689451f793e1bbc
I20260620 07:03:30.348563 13908 ts_tablet_manager.cc:1918] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:30.348678 13908 log.cc:1196] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/4f7a8c9d6a194f08a689451f793e1bbc
I20260620 07:03:30.349076 13907 ts_tablet_manager.cc:1939] T 4f7a8c9d6a194f08a689451f793e1bbc P 84291adf717b4993a42ad9bf8e0b4438: Deleting consensus metadata
I20260620 07:03:30.349166 13908 ts_tablet_manager.cc:1939] T 4f7a8c9d6a194f08a689451f793e1bbc P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting consensus metadata
I20260620 07:03:30.349202 13906 ts_tablet_manager.cc:1939] T 4f7a8c9d6a194f08a689451f793e1bbc P 359f3d3238a24f75860f53dbed3a52de: Deleting consensus metadata
I20260620 07:03:30.350324 13307 catalog_manager.cc:4841] TS 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687): tablet 4f7a8c9d6a194f08a689451f793e1bbc (table test-table [id=57bac543d158419f9079e139e56e3189]) successfully deleted
I20260620 07:03:30.350324 13308 catalog_manager.cc:4841] TS f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855): tablet 4f7a8c9d6a194f08a689451f793e1bbc (table test-table [id=57bac543d158419f9079e139e56e3189]) successfully deleted
I20260620 07:03:30.350669 13307 catalog_manager.cc:4841] TS 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077): tablet 4f7a8c9d6a194f08a689451f793e1bbc (table test-table [id=57bac543d158419f9079e139e56e3189]) successfully deleted
Jun 20 07:03:30 dist-test-slave-wvz3 krb5kdc[13278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:03:30 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:30.387948 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.373967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.11.81.254:41401)
0620 07:03:30.374551 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.374557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.374644 (+ 87us) server_negotiation.cc:407] Connection header received
0620 07:03:30.374833 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.374837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.374902 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.375016 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:30.376353 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.377100 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.378379 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.378842 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.378951 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.382691 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:30.382696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:30.382700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:30.382737 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:30.385379 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:30.386048 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:30.386052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:30.386054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:30.386164 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:30.386591 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:30.386594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:30.386596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:30.386874 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:30.387053 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.387394 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.387612 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":239,"threads_started":1}
I20260620 07:03:30.391356 13323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:30.392033 13323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:30.398550 13600 tablet_service.cc:1467] Processing CreateTablet for tablet 5152ec3062254058b78f2be052f91289 (DEFAULT_TABLE table=test-table [id=7776943d3ffd42f196a264c0536228cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.398541 13460 tablet_service.cc:1467] Processing CreateTablet for tablet 5152ec3062254058b78f2be052f91289 (DEFAULT_TABLE table=test-table [id=7776943d3ffd42f196a264c0536228cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.398782 13739 tablet_service.cc:1467] Processing CreateTablet for tablet 5152ec3062254058b78f2be052f91289 (DEFAULT_TABLE table=test-table [id=7776943d3ffd42f196a264c0536228cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.399032 13600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152ec3062254058b78f2be052f91289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.399039 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152ec3062254058b78f2be052f91289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.399155 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152ec3062254058b78f2be052f91289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.405535 13917 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Bootstrap starting.
I20260620 07:03:30.406282 13918 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Bootstrap starting.
I20260620 07:03:30.406844 13919 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Bootstrap starting.
I20260620 07:03:30.407415 13917 tablet_bootstrap.cc:654] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.408496 13919 tablet_bootstrap.cc:654] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.408864 13918 tablet_bootstrap.cc:654] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.410820 13919 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: No bootstrap required, opened a new log
I20260620 07:03:30.410895 13917 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: No bootstrap required, opened a new log
I20260620 07:03:30.411000 13918 tablet_bootstrap.cc:492] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: No bootstrap required, opened a new log
I20260620 07:03:30.411017 13917 ts_tablet_manager.cc:1397] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 07:03:30.410977 13919 ts_tablet_manager.cc:1397] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:03:30.411170 13918 ts_tablet_manager.cc:1397] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:30.411841 13918 raft_consensus.cc:348] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.411818 13919 raft_consensus.cc:348] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.411818 13917 raft_consensus.cc:348] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.412072 13918 raft_consensus.cc:374] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.412079 13919 raft_consensus.cc:374] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.412079 13917 raft_consensus.cc:374] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.412165 13917 raft_consensus.cc:729] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.412165 13919 raft_consensus.cc:729] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.412166 13918 raft_consensus.cc:729] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.412384 13919 consensus_queue.cc:260] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.412384 13917 consensus_queue.cc:260] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.412479 13918 consensus_queue.cc:260] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.412926 13917 ts_tablet_manager.cc:1428] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:03:30.413571 13918 ts_tablet_manager.cc:1428] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:03:30.413846 13919 ts_tablet_manager.cc:1428] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:30.446058 13923 raft_consensus.cc:482] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:30.446297 13923 raft_consensus.cc:504] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.447923 13923 leader_election.cc:288] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077), 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687)
I20260620 07:03:30.469710 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.448852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36687 (local address 127.11.81.195:44675)
0620 07:03:30.462812 (+ 13960us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:30.462833 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:30.463013 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:30.463579 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:30.463584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:30.463613 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:30.463914 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.463925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.465332 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.465337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.466794 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.466811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.468582 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.468592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.468913 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.469045 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 07:03:30.469171 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:03:30.469333 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13432,"mutex_wait_us":53,"thread_start_us":13237,"threads_started":1}
I20260620 07:03:30.469748 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.449236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:44675 (local address 127.11.81.194:36687)
0620 07:03:30.462769 (+ 13533us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.462773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.462945 (+ 172us) server_negotiation.cc:407] Connection header received
0620 07:03:30.463233 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.463240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.463354 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.463468 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:30.464107 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.465195 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.466969 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.468281 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.468509 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.468589 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.469221 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.469426 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13404,"thread_start_us":13258,"threads_started":1}
I20260620 07:03:30.469755 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.448278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:42193 (local address 127.11.81.193:33077)
0620 07:03:30.462866 (+ 14588us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.462872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.462898 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:30.462973 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.462978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.463046 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.463170 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:30.463703 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.464775 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.467333 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.468500 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.468660 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.468724 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.469261 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.469433 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14464,"thread_start_us":14302,"threads_started":1}
I20260620 07:03:30.470377 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.448174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.11.81.195:42193)
0620 07:03:30.448748 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:30.448768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:30.448889 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:30.463184 (+ 14295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:30.463188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:30.463212 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:30.463488 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.463501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.464945 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.464950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.467130 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.467146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.468747 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.468754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.468984 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.469080 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 07:03:30.469171 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:03:30.470068 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":803,"thread_start_us":272,"threads_started":1}
I20260620 07:03:30.470575 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152ec3062254058b78f2be052f91289" candidate_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438" is_pre_election: true
I20260620 07:03:30.470813 13620 raft_consensus.cc:2454] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4de2f58f06f4bd3bfd9593c992e04ea in term 0.
I20260620 07:03:30.470950 13481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152ec3062254058b78f2be052f91289" candidate_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de" is_pre_election: true
I20260620 07:03:30.471199 13481 raft_consensus.cc:2454] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4de2f58f06f4bd3bfd9593c992e04ea in term 0.
I20260620 07:03:30.471379 13690 leader_election.cc:302] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84291adf717b4993a42ad9bf8e0b4438, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:30.471820 13923 raft_consensus.cc:2781] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:30.471940 13923 raft_consensus.cc:482] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:30.472043 13923 raft_consensus.cc:3037] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.474318 13923 raft_consensus.cc:504] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.474881 13923 leader_election.cc:288] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [CANDIDATE]: Term 1 election: Requested vote from peers 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077), 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687)
I20260620 07:03:30.475507 13481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152ec3062254058b78f2be052f91289" candidate_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359f3d3238a24f75860f53dbed3a52de"
I20260620 07:03:30.475560 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152ec3062254058b78f2be052f91289" candidate_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438"
I20260620 07:03:30.475728 13481 raft_consensus.cc:3037] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.475775 13620 raft_consensus.cc:3037] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.478624 13620 raft_consensus.cc:2454] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de2f58f06f4bd3bfd9593c992e04ea in term 1.
I20260620 07:03:30.478624 13481 raft_consensus.cc:2454] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de2f58f06f4bd3bfd9593c992e04ea in term 1.
I20260620 07:03:30.479068 13690 leader_election.cc:302] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84291adf717b4993a42ad9bf8e0b4438, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:30.479434 13923 raft_consensus.cc:2781] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:30.479907 13923 raft_consensus.cc:686] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 LEADER]: Becoming Leader. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Running, Role: LEADER
I20260620 07:03:30.480351 13923 consensus_queue.cc:237] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.483878 13323 catalog_manager.cc:5495] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea reported cstate change: term changed from 0 to 1, leader changed from <none> to f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195). New cstate: current_term: 1 leader_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } health_report { overall_health: HEALTHY } } }
I20260620 07:03:30.518930 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.513249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.11.81.195:44855)
0620 07:03:30.513876 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.513880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.513899 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:03:30.513987 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.513990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.514068 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.514211 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:30.514606 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.515450 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.516746 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.517160 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.517263 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.517707 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.518298 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.518445 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.518540 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.518619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":379,"threads_started":1}
I20260620 07:03:30.533722 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.528677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:51205 (local address 127.11.81.193:33077)
0620 07:03:30.528841 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.528846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.529355 (+ 509us) server_negotiation.cc:407] Connection header received
0620 07:03:30.529520 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.529524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.529591 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.529662 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:30.530056 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.530781 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.532133 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.532592 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.532692 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.533061 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.533160 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.533219 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.533385 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.533450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:03:30.533840 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.528613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.11.81.195:51205)
0620 07:03:30.529274 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:30.529290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:30.529399 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:30.529689 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:30.529692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:30.529702 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:30.529948 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.529957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.530885 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.530888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.531986 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.531997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.532742 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.532747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.532872 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.532922 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.533278 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.533290 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:03:30.533367 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:03:30.533502 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":331,"threads_started":1}
I20260620 07:03:30.542125 13620 raft_consensus.cc:1261] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Refusing update from remote peer f4de2f58f06f4bd3bfd9593c992e04ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:30.542265 13481 raft_consensus.cc:1261] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Refusing update from remote peer f4de2f58f06f4bd3bfd9593c992e04ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:30.542838 13928 consensus_queue.cc:1035] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:30.543220 13923 consensus_queue.cc:1035] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:30.568960 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.563501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:53195 (local address 127.11.81.195:44855)
0620 07:03:30.563664 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.563670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.564013 (+ 343us) server_negotiation.cc:407] Connection header received
0620 07:03:30.564166 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.564169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.564230 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.564308 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:30.564712 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.565407 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.567031 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.567639 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.567753 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.568285 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.568423 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.568499 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.568612 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.568686 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:03:30.569160 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.563394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44855 (local address 127.11.81.193:53195)
0620 07:03:30.563918 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:30.563933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:30.564065 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:30.564339 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:30.564344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:30.564356 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:30.564583 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.564593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.565526 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.565532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.566896 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.566913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.567884 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.567891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.568077 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.568126 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.568512 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.568524 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:03:30.568594 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:03:30.568728 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260620 07:03:30.596000 13323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:30.596225 13323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:30.599714 13323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Sending DeleteTablet for 3 replicas of tablet 5152ec3062254058b78f2be052f91289
I20260620 07:03:30.600632 13739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5152ec3062254058b78f2be052f91289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 07:03:30.600675 13600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5152ec3062254058b78f2be052f91289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46164
I20260620 07:03:30.600921 13908 tablet_replica.cc:330] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: stopping tablet replica
I20260620 07:03:30.600920 13907 tablet_replica.cc:330] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: stopping tablet replica
I20260620 07:03:30.600967 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet 5152ec3062254058b78f2be052f91289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:30 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38612
I20260620 07:03:30.601125 13908 raft_consensus.cc:2229] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:30.601118 13907 raft_consensus.cc:2229] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:30.601298 13907 raft_consensus.cc:2258] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.601574 13906 tablet_replica.cc:330] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: stopping tablet replica
I20260620 07:03:30.601624 13908 raft_consensus.cc:2258] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.601769 13906 raft_consensus.cc:2229] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:30.601935 13906 raft_consensus.cc:2258] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:30.602844 13906 ts_tablet_manager.cc:1905] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.602938 13908 ts_tablet_manager.cc:1905] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.603565 13323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:30.604245 13323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:30.606741 13907 ts_tablet_manager.cc:1905] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:30.610028 13908 ts_tablet_manager.cc:1918] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:30.610191 13908 log.cc:1196] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/5152ec3062254058b78f2be052f91289
I20260620 07:03:30.610493 13906 ts_tablet_manager.cc:1918] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:30.610635 13906 log.cc:1196] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/5152ec3062254058b78f2be052f91289
I20260620 07:03:30.610662 13908 ts_tablet_manager.cc:1939] T 5152ec3062254058b78f2be052f91289 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting consensus metadata
I20260620 07:03:30.611006 13906 ts_tablet_manager.cc:1939] T 5152ec3062254058b78f2be052f91289 P 359f3d3238a24f75860f53dbed3a52de: Deleting consensus metadata
I20260620 07:03:30.612191 13307 catalog_manager.cc:4841] TS 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077): tablet 5152ec3062254058b78f2be052f91289 (table test-table [id=7776943d3ffd42f196a264c0536228cb]) successfully deleted
I20260620 07:03:30.612840 13907 ts_tablet_manager.cc:1918] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:03:30.612985 13907 log.cc:1196] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/5152ec3062254058b78f2be052f91289
I20260620 07:03:30.612998 13308 catalog_manager.cc:4841] TS f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855): tablet 5152ec3062254058b78f2be052f91289 (table test-table [id=7776943d3ffd42f196a264c0536228cb]) successfully deleted
I20260620 07:03:30.614531 13907 ts_tablet_manager.cc:1939] T 5152ec3062254058b78f2be052f91289 P 84291adf717b4993a42ad9bf8e0b4438: Deleting consensus metadata
I20260620 07:03:30.616114 13307 catalog_manager.cc:4841] TS 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687): tablet 5152ec3062254058b78f2be052f91289 (table test-table [id=7776943d3ffd42f196a264c0536228cb]) successfully deleted
I20260620 07:03:30.616585 13600 tablet_service.cc:1467] Processing CreateTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 (DEFAULT_TABLE table=test-table [id=786ba5c488ef4b90b1546db5939a7513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.616684 13460 tablet_service.cc:1467] Processing CreateTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 (DEFAULT_TABLE table=test-table [id=786ba5c488ef4b90b1546db5939a7513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.617002 13600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22dbc81385d4a578c7abfbe13dc6205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.617139 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22dbc81385d4a578c7abfbe13dc6205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.619980 13739 tablet_service.cc:1467] Processing CreateTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 (DEFAULT_TABLE table=test-table [id=786ba5c488ef4b90b1546db5939a7513]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:30.620419 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22dbc81385d4a578c7abfbe13dc6205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:30.622098 13918 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Bootstrap starting.
I20260620 07:03:30.623634 13917 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Bootstrap starting.
I20260620 07:03:30.623762 13918 tablet_bootstrap.cc:654] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.625319 13917 tablet_bootstrap.cc:654] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.626971 13919 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Bootstrap starting.
I20260620 07:03:30.628407 13919 tablet_bootstrap.cc:654] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:30.634122 13918 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: No bootstrap required, opened a new log
I20260620 07:03:30.634358 13918 ts_tablet_manager.cc:1397] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260620 07:03:30.635226 13917 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: No bootstrap required, opened a new log
I20260620 07:03:30.635188 13918 raft_consensus.cc:348] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.635530 13918 raft_consensus.cc:374] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.635620 13918 raft_consensus.cc:729] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84291adf717b4993a42ad9bf8e0b4438, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.635735 13917 ts_tablet_manager.cc:1397] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 07:03:30.635890 13918 consensus_queue.cc:260] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.636669 13919 tablet_bootstrap.cc:492] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: No bootstrap required, opened a new log
I20260620 07:03:30.636847 13919 ts_tablet_manager.cc:1397] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 07:03:30.636698 13917 raft_consensus.cc:348] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.637117 13917 raft_consensus.cc:374] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.637244 13917 raft_consensus.cc:729] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.637579 13917 consensus_queue.cc:260] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.637601 13919 raft_consensus.cc:348] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.637844 13919 raft_consensus.cc:374] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:30.637933 13919 raft_consensus.cc:729] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de2f58f06f4bd3bfd9593c992e04ea, State: Initialized, Role: FOLLOWER
I20260620 07:03:30.638201 13919 consensus_queue.cc:260] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.638728 13918 ts_tablet_manager.cc:1428] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 07:03:30.639139 13919 ts_tablet_manager.cc:1428] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 07:03:30.639616 13917 ts_tablet_manager.cc:1428] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 07:03:30.644809 13888 raft_consensus.cc:482] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:30.645035 13888 raft_consensus.cc:504] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.645820 13888 leader_election.cc:288] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:30.646250 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22dbc81385d4a578c7abfbe13dc6205" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" is_pre_election: true
I20260620 07:03:30.646283 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22dbc81385d4a578c7abfbe13dc6205" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438" is_pre_election: true
I20260620 07:03:30.646517 13758 raft_consensus.cc:2454] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 0.
I20260620 07:03:30.646528 13620 raft_consensus.cc:2454] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 0.
I20260620 07:03:30.646953 13411 leader_election.cc:302] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, 84291adf717b4993a42ad9bf8e0b4438; no voters:
I20260620 07:03:30.647310 13888 raft_consensus.cc:2781] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:30.647432 13888 raft_consensus.cc:482] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:30.647529 13888 raft_consensus.cc:3037] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.649948 13888 raft_consensus.cc:504] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.650622 13888 leader_election.cc:288] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 election: Requested vote from peers 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687), f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855)
I20260620 07:03:30.651072 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22dbc81385d4a578c7abfbe13dc6205" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea"
I20260620 07:03:30.651321 13758 raft_consensus.cc:3037] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.651157 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22dbc81385d4a578c7abfbe13dc6205" candidate_uuid: "359f3d3238a24f75860f53dbed3a52de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84291adf717b4993a42ad9bf8e0b4438"
I20260620 07:03:30.651557 13620 raft_consensus.cc:3037] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:30.653555 13758 raft_consensus.cc:2454] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 1.
I20260620 07:03:30.653949 13412 leader_election.cc:302] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359f3d3238a24f75860f53dbed3a52de, f4de2f58f06f4bd3bfd9593c992e04ea; no voters:
I20260620 07:03:30.654217 13888 raft_consensus.cc:2781] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:30.654364 13888 raft_consensus.cc:686] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 1 LEADER]: Becoming Leader. State: Replica: 359f3d3238a24f75860f53dbed3a52de, State: Running, Role: LEADER
I20260620 07:03:30.654621 13888 consensus_queue.cc:237] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } }
I20260620 07:03:30.654922 13620 raft_consensus.cc:2454] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359f3d3238a24f75860f53dbed3a52de in term 1.
I20260620 07:03:30.657401 13322 catalog_manager.cc:5495] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de reported cstate change: term changed from 0 to 1, leader changed from <none> to 359f3d3238a24f75860f53dbed3a52de (127.11.81.193). New cstate: current_term: 1 leader_uuid: "359f3d3238a24f75860f53dbed3a52de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359f3d3238a24f75860f53dbed3a52de" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 } health_report { overall_health: UNKNOWN } } }
I20260620 07:03:30.684267 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.678726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.11.81.193:33077)
0620 07:03:30.678931 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.678935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.678952 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:30.679021 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.679027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.679110 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.679185 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:30.679742 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.680500 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.682439 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.682856 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.682970 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.683567 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.683687 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.683765 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.683911 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.683988 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:03:30.697611 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.693193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:51209 (local address 127.11.81.193:33077)
0620 07:03:30.693349 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:30.693353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:30.693370 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:30.693460 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:30.693464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:30.693527 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:30.693594 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:30.693938 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.694640 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.695900 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.696316 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.696422 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.696794 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.696931 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.697018 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:30.697186 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 07:03:30.697254 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":70}
I20260620 07:03:30.697732 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:30.692949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33077 (local address 127.11.81.193:51209)
0620 07:03:30.693163 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:30.693203 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:30.693344 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:30.693613 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:30.693616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:30.693627 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:03:30.693819 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.693828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.694758 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.694764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.695773 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:30.695786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:30.696449 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:30.696455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:30.696592 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:30.696629 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:03:30.697048 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:03:30.697064 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 07:03:30.697168 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:03:30.697332 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":19}
I20260620 07:03:30.706889 13620 raft_consensus.cc:1261] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Refusing update from remote peer 359f3d3238a24f75860f53dbed3a52de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:30.706984 13758 raft_consensus.cc:1261] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Refusing update from remote peer 359f3d3238a24f75860f53dbed3a52de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:30.707499 13888 consensus_queue.cc:1035] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Connected to new peer: Peer: permanent_uuid: "84291adf717b4993a42ad9bf8e0b4438" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:30.707791 13930 consensus_queue.cc:1035] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4de2f58f06f4bd3bfd9593c992e04ea" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:03:30.732589 13886 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:03:32.328394 13321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:32.328666 13321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:32.332563 13321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52905da934194211a32b23bd3d8cd0a6: Sending DeleteTablet for 3 replicas of tablet c22dbc81385d4a578c7abfbe13dc6205
I20260620 07:03:32.333310 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:32 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38612
I20260620 07:03:32.333417 13739 tablet_service.cc:1514] Processing DeleteTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:32 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 07:03:32.333604 13600 tablet_service.cc:1514] Processing DeleteTablet for tablet c22dbc81385d4a578c7abfbe13dc6205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:32 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46164
I20260620 07:03:32.334154 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13395
I20260620 07:03:32.334652 13969 tablet_replica.cc:330] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: stopping tablet replica
I20260620 07:03:32.334882 13969 raft_consensus.cc:2229] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:32.335196 13971 tablet_replica.cc:330] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: stopping tablet replica
I20260620 07:03:32.335455 13971 raft_consensus.cc:2229] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:32.335518 13970 tablet_replica.cc:330] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: stopping tablet replica
I20260620 07:03:32.335501 13969 raft_consensus.cc:2258] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:32.335685 13970 raft_consensus.cc:2229] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:32.335814 13971 raft_consensus.cc:2258] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:32.335950 13970 raft_consensus.cc:2258] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:32.336485 13969 ts_tablet_manager.cc:1905] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:32.337222 13971 ts_tablet_manager.cc:1905] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:32.343362 13970 ts_tablet_manager.cc:1905] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:32.343909 13969 ts_tablet_manager.cc:1918] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:32.344125 13969 log.cc:1196] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/c22dbc81385d4a578c7abfbe13dc6205
I20260620 07:03:32.344594 13969 ts_tablet_manager.cc:1939] T c22dbc81385d4a578c7abfbe13dc6205 P 359f3d3238a24f75860f53dbed3a52de: Deleting consensus metadata
I20260620 07:03:32.345278 13971 ts_tablet_manager.cc:1918] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:32.345806 13307 catalog_manager.cc:4841] TS 359f3d3238a24f75860f53dbed3a52de (127.11.81.193:33077): tablet c22dbc81385d4a578c7abfbe13dc6205 (table test-table [id=786ba5c488ef4b90b1546db5939a7513]) successfully deleted
I20260620 07:03:32.346221 13971 log.cc:1196] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/c22dbc81385d4a578c7abfbe13dc6205
I20260620 07:03:32.346675 13971 ts_tablet_manager.cc:1939] T c22dbc81385d4a578c7abfbe13dc6205 P 84291adf717b4993a42ad9bf8e0b4438: Deleting consensus metadata
I20260620 07:03:32.347739 13307 catalog_manager.cc:4841] TS 84291adf717b4993a42ad9bf8e0b4438 (127.11.81.194:36687): tablet c22dbc81385d4a578c7abfbe13dc6205 (table test-table [id=786ba5c488ef4b90b1546db5939a7513]) successfully deleted
I20260620 07:03:32.350416 13970 ts_tablet_manager.cc:1918] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:03:32.350567 13970 log.cc:1196] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/c22dbc81385d4a578c7abfbe13dc6205
I20260620 07:03:32.351001 13970 ts_tablet_manager.cc:1939] T c22dbc81385d4a578c7abfbe13dc6205 P f4de2f58f06f4bd3bfd9593c992e04ea: Deleting consensus metadata
I20260620 07:03:32.352195 13308 catalog_manager.cc:4841] TS f4de2f58f06f4bd3bfd9593c992e04ea (127.11.81.195:44855): tablet c22dbc81385d4a578c7abfbe13dc6205 (table test-table [id=786ba5c488ef4b90b1546db5939a7513]) successfully deleted
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:32.362964 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:32.336489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.11.81.193:33077)
0620 07:03:32.347720 (+ 11231us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:32.347726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:32.347745 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:03:32.347800 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:32.347805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:32.347882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:32.349594 (+ 1712us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:32.349758 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.350791 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.352827 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.353237 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.353353 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:32.357735 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:32.357751 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:32.357768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:32.357809 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:32.360570 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.361221 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.361229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.361231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.361384 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.361726 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.361729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.361731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.362086 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:32.362305 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:32.362528 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:03:32.362703 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11094,"thread_start_us":10994,"threads_started":1}
W20260620 07:03:32.363715 13526 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56728
W20260620 07:03:32.364341 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13395: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:32.364578 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13395
I20260620 07:03:32.386646 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13535
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:32.402184 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:32.387419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.11.81.194:36687)
0620 07:03:32.387908 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:32.387912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:32.387928 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:03:32.387986 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:32.387989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:32.388048 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:32.388141 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:32.389646 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.390442 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.391949 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.392377 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.392538 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:32.396648 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:32.396663 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:32.396680 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:32.396723 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:32.399609 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.400351 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.400360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.400362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.400576 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.400907 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.400911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.400914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.401289 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:32.401579 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:32.401727 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 07:03:32.401913 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
W20260620 07:03:32.402854 13665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42038
W20260620 07:03:32.403475 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13535: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:32.403610 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13535
I20260620 07:03:32.420735 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13674
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939010, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:32.436072 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:32.421457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.11.81.195:44855)
0620 07:03:32.421973 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:32.421978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:32.421999 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:03:32.422054 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:32.422058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:32.422118 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:32.422213 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:32.423719 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.424482 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.425755 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:32.426210 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:32.426314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:32.430510 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:32.430527 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:32.430544 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:32.430580 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:32.433275 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.434022 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.434029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.434031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.434150 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:32.434540 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:32.434545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:32.434548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:32.434915 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:32.435211 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:32.435506 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 07:03:32.435724 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":281,"threads_started":1}
W20260620 07:03:32.436697 13804 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36426
W20260620 07:03:32.437286 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:32.437444 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13674
I20260620 07:03:32.454308 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13291
I20260620 07:03:32.455624 13363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:32.570538 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13291
2026-06-20T07:03:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7833 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13979](info): set up 2 sockets
Jun 20 07:03:32 dist-test-slave-wvz3 krb5kdc[13979](info): commencing operation
krb5kdc: starting...
W20260620 07:03:33.571252 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:33 dist-test-slave-wvz3 krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:33.761289 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38469
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:34.078267 13992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:34.078547 13992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:34.078692 13992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:34.086148 13992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:34.086273 13992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:34.086321 13992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:34.086398 13992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:34.086457 13992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:34.096881 13992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:34.098209 13992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:34.099792 13992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:34.112200 13997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:34.112249 13998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:34.112202 14000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:34.112833 13992 server_base.cc:1034] running on GCE node
I20260620 07:03:34.113775 13992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:34.115309 13992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:34.116556 13992 hybrid_clock.cc:648] HybridClock initialized: now 1781939014116510 us; error 35 us; skew 500 ppm
I20260620 07:03:34.116935 13992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:34 dist-test-slave-wvz3 krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939014, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:34.126675 13992 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:34.128445 13992 webserver.cc:460] Webserver started at http://127.11.81.254:36761/ using document root <none> and password file <none>
I20260620 07:03:34.129190 13992 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:34.129307 13992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:34.129634 13992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:34.132783 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "70ebdb47015341458027a12863899e2e"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "d70ea62a5f09d36792597f0bdf495908"
server_key_iv: "2f90aad636fa59e0262aa767a40d3c5e"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.133579 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "70ebdb47015341458027a12863899e2e"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "d70ea62a5f09d36792597f0bdf495908"
server_key_iv: "2f90aad636fa59e0262aa767a40d3c5e"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.139276 13992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:34.143074 14007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:34.144857 13992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:34.145047 13992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "70ebdb47015341458027a12863899e2e"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "d70ea62a5f09d36792597f0bdf495908"
server_key_iv: "2f90aad636fa59e0262aa767a40d3c5e"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.145257 13992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:34.164130 13992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:34.168653 13992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:34.168946 13992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:34.186802 13992 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:38469
I20260620 07:03:34.186808 14058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:38469 every 8 connection(s)
I20260620 07:03:34.188299 13992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:34.189747 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 13992
I20260620 07:03:34.189971 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:34.190866 11591 external_mini_cluster.cc:1428] Setting key fd248c007523f94db8735521f5637322
I20260620 07:03:34.193361 14059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:03:34 dist-test-slave-wvz3 krb5kdc[13979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:34.207209 14059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e: Bootstrap starting.
I20260620 07:03:34.210628 14059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:34.211802 14059 log.cc:824] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:34.215395 14059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e: No bootstrap required, opened a new log
I20260620 07:03:34.215636 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:34.192980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.11.81.254:38469)
0620 07:03:34.193784 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:34.193797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:34.193841 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:03:34.195555 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:34.195580 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:34.196017 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:34.196403 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:34.198310 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.199877 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.201632 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.202404 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.202572 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:34.207631 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:34.207641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:34.207659 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:34.207708 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:34.210924 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:34.211773 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:34.211780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:34.211791 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:34.211993 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:34.212489 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:34.212494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:34.212497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:34.213299 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:34.213529 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:34.214065 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 07:03:34.214455 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":348,"threads_started":1}
I20260620 07:03:34.223059 14059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } }
I20260620 07:03:34.223516 14059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:34.223621 14059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ebdb47015341458027a12863899e2e, State: Initialized, Role: FOLLOWER
I20260620 07:03:34.224242 14059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } }
I20260620 07:03:34.224473 14059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:34.224618 14059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:34.224776 14059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:34.226970 14059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } }
I20260620 07:03:34.227563 14059 leader_election.cc:302] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ebdb47015341458027a12863899e2e; no voters:
I20260620 07:03:34.228176 14059 leader_election.cc:288] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:34.228318 14064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:34.229091 14064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [term 1 LEADER]: Becoming Leader. State: Replica: 70ebdb47015341458027a12863899e2e, State: Running, Role: LEADER
I20260620 07:03:34.229627 14064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } }
I20260620 07:03:34.230212 14059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:34.232470 14065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ebdb47015341458027a12863899e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } } }
I20260620 07:03:34.232901 14065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:34.233242 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ebdb47015341458027a12863899e2e. Latest consensus state: current_term: 1 leader_uuid: "70ebdb47015341458027a12863899e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ebdb47015341458027a12863899e2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38469 } } }
I20260620 07:03:34.233548 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:34.235850 14069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:34.241436 14069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:34.251081 14069 catalog_manager.cc:1347] Generated new cluster ID: d2ae8773e07a4de5932a51020cbc907e
I20260620 07:03:34.251211 14069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:34.280584 14069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:34.281993 14069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:34.294953 14069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ebdb47015341458027a12863899e2e: Generated new TSK 0
I20260620 07:03:34.295984 14069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:34.340756 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:34.663740 14085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:34.664052 14085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:34.664149 14085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:34.671639 14085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:34.671761 14085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:34.671873 14085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:34.682303 14085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:34.683622 14085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:34.684921 14085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:34.698292 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:34.698331 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:34.698539 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:34.698979 14085 server_base.cc:1034] running on GCE node
I20260620 07:03:34.699641 14085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:34.700662 14085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:34.701881 14085 hybrid_clock.cc:648] HybridClock initialized: now 1781939014701817 us; error 55 us; skew 500 ppm
I20260620 07:03:34.702198 14085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:34 dist-test-slave-wvz3 krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939014, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:34.705449 14085 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:34.707023 14085 webserver.cc:460] Webserver started at http://127.11.81.193:41373/ using document root <none> and password file <none>
I20260620 07:03:34.707837 14085 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:34.707953 14085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:34.708264 14085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:34.711274 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "71e89d57ea3542c9bdab1ea53f31ffa1"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "35339924fa5c2a05f4710a7b40d75f95"
server_key_iv: "6ae45492ce24273ea30ad722a082c733"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.712184 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "71e89d57ea3542c9bdab1ea53f31ffa1"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "35339924fa5c2a05f4710a7b40d75f95"
server_key_iv: "6ae45492ce24273ea30ad722a082c733"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.717921 14085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:34.721566 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:34.723028 14085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:03:34.723197 14085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "71e89d57ea3542c9bdab1ea53f31ffa1"
format_stamp: "Formatted at 2026-06-20 07:03:34 on dist-test-slave-wvz3"
server_key: "35339924fa5c2a05f4710a7b40d75f95"
server_key_iv: "6ae45492ce24273ea30ad722a082c733"
server_key_version: "encryptionkey@0"
I20260620 07:03:34.723436 14085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:34.748292 14085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:34.752684 14085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:34.752957 14085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:34.753911 14085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:34.755276 14085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:34.755394 14085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:34.755518 14085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:34.755578 14085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:34.784457 14085 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:46647
I20260620 07:03:34.784510 14212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:46647 every 8 connection(s)
I20260620 07:03:34.785733 14085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:34.789999 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14085
I20260620 07:03:34.790218 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:34.790963 11591 external_mini_cluster.cc:1428] Setting key 1f19b30ed076002fde5b20516afd75bf
Jun 20 07:03:34 dist-test-slave-wvz3 krb5kdc[13979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939014, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:34.808661 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:34.788325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:48189 (local address 127.11.81.254:38469)
0620 07:03:34.788749 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:34.788754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:34.789882 (+ 1128us) server_negotiation.cc:407] Connection header received
0620 07:03:34.792222 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:34.792227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:34.792300 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:34.792414 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:34.794266 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.795046 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.797263 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.797831 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.797925 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:34.802158 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:34.802163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:34.802167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:34.802203 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:34.804766 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:34.805663 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:34.805667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:34.805669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:34.805802 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:34.806500 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:34.806504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:34.806506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:34.806815 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:34.806948 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:34.808020 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0620 07:03:34.808279 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":219,"threads_started":1}
I20260620 07:03:34.809437 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:34.788912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38469 (local address 127.11.81.193:48189)
0620 07:03:34.789671 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:34.789734 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:34.791954 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:34.792655 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:34.792664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:34.793282 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:34.794082 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:34.794106 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.795228 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.795233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:34.797082 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:34.797097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:34.798257 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:34.798263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:34.798550 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:34.799336 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:34.799364 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:34.801883 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:34.804935 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:34.804946 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:34.804962 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:34.805514 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:34.806005 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:34.806011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:34.806015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:34.806318 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:34.806969 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:34.806978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:34.807544 (+ 566us) client_negotiation.cc:769] Sending connection context
0620 07:03:34.807819 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 07:03:34.808266 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:34.811256 14213 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38469
I20260620 07:03:34.811595 14213 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:03:34.812362 14213 heartbeater.cc:511] Master 127.11.81.254:38469 requested a full tablet report, sending...
I20260620 07:03:34.814520 14024 ts_manager.cc:194] Registered new tserver with Master: 71e89d57ea3542c9bdab1ea53f31ffa1 (127.11.81.193:46647)
I20260620 07:03:34.816035 14024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:48189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:34.835872 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:35.154764 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:35.155076 14219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:35.155257 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:35.162647 14219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:35.162781 14219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:35.162946 14219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:35.172868 14219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:35.174201 14219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:35.175627 14219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:35.188781 14225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:35.188807 14224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:35.189458 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:35.190011 14219 server_base.cc:1034] running on GCE node
I20260620 07:03:35.190676 14219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:35.191753 14219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:35.192957 14219 hybrid_clock.cc:648] HybridClock initialized: now 1781939015192898 us; error 57 us; skew 500 ppm
I20260620 07:03:35.193301 14219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:35 dist-test-slave-wvz3 krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939015, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:35.196822 14219 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:35.198872 14219 webserver.cc:460] Webserver started at http://127.11.81.194:38561/ using document root <none> and password file <none>
I20260620 07:03:35.199645 14219 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:35.199765 14219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:35.200081 14219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:35.203131 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "889b7c9dfe7f43d799edfa0914ed1764"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "7a32cc9e6a2454444d4181bc3bf785b2"
server_key_iv: "6f1b1b5afab63154a981c63cbd70bb90"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.204041 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "889b7c9dfe7f43d799edfa0914ed1764"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "7a32cc9e6a2454444d4181bc3bf785b2"
server_key_iv: "6f1b1b5afab63154a981c63cbd70bb90"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.209699 14219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:35.214248 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.215888 14219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:03:35.216077 14219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "889b7c9dfe7f43d799edfa0914ed1764"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "7a32cc9e6a2454444d4181bc3bf785b2"
server_key_iv: "6f1b1b5afab63154a981c63cbd70bb90"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.216295 14219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:35.236593 14219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:35.241171 14219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:35.241478 14219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:35.242560 14219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:35.244110 14219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:35.244195 14219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.244314 14219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:35.244364 14219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.276981 14219 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:46307
I20260620 07:03:35.277033 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:46307 every 8 connection(s)
I20260620 07:03:35.278221 14219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:35.284337 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14219
I20260620 07:03:35.284548 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:35.284991 11591 external_mini_cluster.cc:1428] Setting key 5018e6b4400e7e6e676bab9611ddaf98
Jun 20 07:03:35 dist-test-slave-wvz3 krb5kdc[13979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939015, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:35.301061 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:35.280820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:52609 (local address 127.11.81.254:38469)
0620 07:03:35.281049 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:35.281055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:35.282246 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 07:03:35.284111 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:35.284117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:35.284209 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:35.284337 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:35.286168 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.287249 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.289691 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.290105 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.290209 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:35.294387 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:35.294401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:35.294405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:35.294451 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:35.297606 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:35.298505 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:35.298510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:35.298514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:35.298662 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:35.299157 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:35.299162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:35.299165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:35.299565 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:35.299707 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:35.300449 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 07:03:35.300688 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:03:35.302039 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:35.281334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38469 (local address 127.11.81.194:52609)
0620 07:03:35.282067 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:35.282113 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:35.283781 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:35.284562 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:35.284572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:35.285279 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:35.285968 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:35.285995 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.287494 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.287499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:35.289509 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:35.289536 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.290341 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.290347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:35.290645 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:35.291405 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:35.291439 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:35.294115 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:35.297806 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:35.297814 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:35.297830 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:35.298362 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:35.298805 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:35.298809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:35.298811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:35.299036 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:35.299717 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:35.299723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:35.300077 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 07:03:35.300327 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:03:35.300874 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
I20260620 07:03:35.303637 14347 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38469
I20260620 07:03:35.303922 14347 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:03:35.304677 14347 heartbeater.cc:511] Master 127.11.81.254:38469 requested a full tablet report, sending...
I20260620 07:03:35.306175 14024 ts_manager.cc:194] Registered new tserver with Master: 889b7c9dfe7f43d799edfa0914ed1764 (127.11.81.194:46307)
I20260620 07:03:35.307034 14024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:52609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:35.331979 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:35.648908 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:35.649215 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:35.649351 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:35.656740 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:35.656888 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:35.657043 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:35.666803 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:35.668303 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:35.669593 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:35.683029 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:35.682941 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:35.683085 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:35.684240 14352 server_base.cc:1034] running on GCE node
I20260620 07:03:35.684886 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:35.685839 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:35.687086 14352 hybrid_clock.cc:648] HybridClock initialized: now 1781939015687025 us; error 45 us; skew 500 ppm
I20260620 07:03:35.687466 14352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:35 dist-test-slave-wvz3 krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939015, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:35.690745 14352 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:35.692337 14352 webserver.cc:460] Webserver started at http://127.11.81.195:40233/ using document root <none> and password file <none>
I20260620 07:03:35.693051 14352 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:35.693153 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:35.693455 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:35.696542 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "9046088967c74d3e9d298a44d7a25edf"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "a74c65ffbf6c3e5c19eb8a7d28c84e4a"
server_key_iv: "b9762eb6930f55cddf0ab8364c59b108"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.697379 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "9046088967c74d3e9d298a44d7a25edf"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "a74c65ffbf6c3e5c19eb8a7d28c84e4a"
server_key_iv: "b9762eb6930f55cddf0ab8364c59b108"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.703029 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:03:35.706964 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.708685 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:35.708840 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "9046088967c74d3e9d298a44d7a25edf"
format_stamp: "Formatted at 2026-06-20 07:03:35 on dist-test-slave-wvz3"
server_key: "a74c65ffbf6c3e5c19eb8a7d28c84e4a"
server_key_iv: "b9762eb6930f55cddf0ab8364c59b108"
server_key_version: "encryptionkey@0"
I20260620 07:03:35.709105 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:35.722877 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:35.727104 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:35.727380 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:35.728411 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:35.729800 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:35.729876 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.729992 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:35.730065 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:35.759855 14352 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36955
I20260620 07:03:35.759877 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36955 every 8 connection(s)
I20260620 07:03:35.761099 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:35.769851 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14352
I20260620 07:03:35.770022 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:35.770423 11591 external_mini_cluster.cc:1428] Setting key 8d664fd59546147633c1a05702e26460
Jun 20 07:03:35 dist-test-slave-wvz3 krb5kdc[13979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939015, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:35.780316 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:35.763643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:59759 (local address 127.11.81.254:38469)
0620 07:03:35.763817 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:35.763821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:35.765070 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 07:03:35.766958 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:35.766963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:35.767055 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:35.767141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:35.768706 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.769428 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.771166 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.771639 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.771750 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:35.775151 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:35.775156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:35.775159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:35.775192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:35.777424 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:35.778099 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:35.778103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:35.778106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:35.778226 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:35.778632 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:35.778637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:35.778639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:35.778903 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:35.779026 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:35.779768 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 07:03:35.779982 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:03:35.781227 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:35.764140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38469 (local address 127.11.81.195:59759)
0620 07:03:35.764869 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:35.764910 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:35.766665 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:35.767432 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:35.767441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:35.767953 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:35.768531 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:35.768548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.769584 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.769588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:35.770984 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:35.770998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:35.771860 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:35.771868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:35.772092 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:35.772751 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:35.772776 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:35.774933 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:35.777552 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:35.777561 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:35.777579 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:35.777982 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:35.778331 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:35.778336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:35.778339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:35.778522 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:35.779026 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:35.779034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:35.779413 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 07:03:35.779655 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 07:03:35.780160 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":280,"threads_started":1}
I20260620 07:03:35.782670 14480 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38469
I20260620 07:03:35.782971 14480 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:35.783743 14480 heartbeater.cc:511] Master 127.11.81.254:38469 requested a full tablet report, sending...
I20260620 07:03:35.785418 14023 ts_manager.cc:194] Registered new tserver with Master: 9046088967c74d3e9d298a44d7a25edf (127.11.81.195:36955)
I20260620 07:03:35.786538 14023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:59759
I20260620 07:03:35.797992 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:35.819059 14213 heartbeater.cc:503] Master 127.11.81.254:38469 was elected leader, sending a full tablet report...
W20260620 07:03:35.828174 14215 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:35.825559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.11.81.254:38469)
0620 07:03:35.825793 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:35.825802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:35.826183 (+ 381us) server_negotiation.cc:407] Connection header received
0620 07:03:35.826335 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:35.826339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:35.826404 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:35.826489 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:35.827897 (+ 1408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47934: BlockingRecv error: recv got EOF from 127.0.0.1:47934 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:03:35.831856 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14085
W20260620 07:03:35.834614 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14085: Not authorized: Client connection negotiation failed: client connection to 127.11.81.193:46647: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:03:35.834812 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14085
I20260620 07:03:35.847612 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14219
W20260620 07:03:35.850421 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14219: Not authorized: Client connection negotiation failed: client connection to 127.11.81.194:46307: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:03:35.850575 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14219
I20260620 07:03:35.862609 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14352
W20260620 07:03:35.865571 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14352: Not authorized: Client connection negotiation failed: client connection to 127.11.81.195:36955: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:03:35.865696 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14352
I20260620 07:03:35.877614 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 13992
I20260620 07:03:35.878978 14054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:35.983081 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 13992
2026-06-20T07:03:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3409 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:36 dist-test-slave-wvz3 krb5kdc[14497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:36 dist-test-slave-wvz3 krb5kdc[14497](info): set up 2 sockets
Jun 20 07:03:36 dist-test-slave-wvz3 krb5kdc[14497](info): commencing operation
krb5kdc: starting...
W20260620 07:03:36.987509 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:37.174855 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43511
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:37.494673 14510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:37.495012 14510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:37.495164 14510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:37.502498 14510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:37.502624 14510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:37.502702 14510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:37.502784 14510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:37.502847 14510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:37.512877 14510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:37.514240 14510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:37.515662 14510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:37.527424 14516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:37.527453 14518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:37.527484 14515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:37.528124 14510 server_base.cc:1034] running on GCE node
I20260620 07:03:37.528918 14510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:37.530326 14510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:37.531540 14510 hybrid_clock.cc:648] HybridClock initialized: now 1781939017531491 us; error 42 us; skew 500 ppm
I20260620 07:03:37.531965 14510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939017, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:37.538506 14510 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:37.540838 14510 webserver.cc:460] Webserver started at http://127.11.81.254:34395/ using document root <none> and password file <none>
I20260620 07:03:37.541720 14510 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:37.541847 14510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:37.542162 14510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:37.545882 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "158943df9665432885a51bd6de15662a"
format_stamp: "Formatted at 2026-06-20 07:03:37 on dist-test-slave-wvz3"
server_key: "2f5dfcf919ae8e127c657225baa6a9b3"
server_key_iv: "620823989b9dde4e847860f81e9a2dfc"
server_key_version: "encryptionkey@0"
I20260620 07:03:37.546666 14510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "158943df9665432885a51bd6de15662a"
format_stamp: "Formatted at 2026-06-20 07:03:37 on dist-test-slave-wvz3"
server_key: "2f5dfcf919ae8e127c657225baa6a9b3"
server_key_iv: "620823989b9dde4e847860f81e9a2dfc"
server_key_version: "encryptionkey@0"
I20260620 07:03:37.553407 14510 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260620 07:03:37.557977 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:37.559674 14510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:37.559867 14510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "158943df9665432885a51bd6de15662a"
format_stamp: "Formatted at 2026-06-20 07:03:37 on dist-test-slave-wvz3"
server_key: "2f5dfcf919ae8e127c657225baa6a9b3"
server_key_iv: "620823989b9dde4e847860f81e9a2dfc"
server_key_version: "encryptionkey@0"
I20260620 07:03:37.560107 14510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:37.577461 14510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:37.581761 14510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:37.582031 14510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:37.599964 14510 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43511
I20260620 07:03:37.599987 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43511 every 8 connection(s)
I20260620 07:03:37.601364 14510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:37.603034 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14510
I20260620 07:03:37.603261 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:37.604080 11591 external_mini_cluster.cc:1428] Setting key 0577d6d33384a438564f580f908c8399
I20260620 07:03:37.606094 14577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:37.619369 14577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a: Bootstrap starting.
I20260620 07:03:37.622694 14577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:37.624032 14577 log.cc:824] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:37.627257 14577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a: No bootstrap required, opened a new log
I20260620 07:03:37.627718 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:37.605954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.11.81.254:43511)
0620 07:03:37.606714 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:37.606727 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:37.606766 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:03:37.608454 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:37.608479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:37.608900 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:37.609269 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:37.611060 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:37.612518 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:37.614258 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:37.615027 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:37.615184 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:37.619986 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:37.619996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:37.620013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:37.620057 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:37.623028 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:37.623921 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:37.623927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:37.623934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:37.624116 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:37.624546 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:37.624554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:37.624558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:37.625394 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:37.625611 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:37.626135 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 07:03:37.626541 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":353,"threads_started":1}
I20260620 07:03:37.631882 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:37.636759 14577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } }
I20260620 07:03:37.637280 14577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:37.637382 14577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158943df9665432885a51bd6de15662a, State: Initialized, Role: FOLLOWER
I20260620 07:03:37.638084 14577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } }
I20260620 07:03:37.638350 14577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:37.638476 14577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:37.638628 14577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:37.641089 14577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } }
I20260620 07:03:37.641635 14577 leader_election.cc:302] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158943df9665432885a51bd6de15662a; no voters:
I20260620 07:03:37.642268 14577 leader_election.cc:288] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:37.642369 14583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:37.643486 14583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [term 1 LEADER]: Becoming Leader. State: Replica: 158943df9665432885a51bd6de15662a, State: Running, Role: LEADER
I20260620 07:03:37.644028 14583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } }
I20260620 07:03:37.644649 14577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:37.646777 14585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158943df9665432885a51bd6de15662a. Latest consensus state: current_term: 1 leader_uuid: "158943df9665432885a51bd6de15662a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } } }
I20260620 07:03:37.647145 14585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:37.647710 14584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "158943df9665432885a51bd6de15662a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158943df9665432885a51bd6de15662a" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43511 } } }
I20260620 07:03:37.647979 14584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:37.655926 14588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:37.661365 14588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:37.672539 14588 catalog_manager.cc:1347] Generated new cluster ID: eadb3f1ab5fb4709b59a9b3ee8a37136
I20260620 07:03:37.672684 14588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:37.681660 14606 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:03:37.691552 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:37.684064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.11.81.254:43511)
0620 07:03:37.684271 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:37.684278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:37.684868 (+ 590us) server_negotiation.cc:407] Connection header received
0620 07:03:37.685055 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:37.685059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:37.685125 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:37.685213 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:37.686352 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:37.687126 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:37.688461 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:37.688880 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:37.689006 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:37.690048 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:37.690053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:37.690056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:37.690720 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:37.690782 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:37.690805 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:37.690895 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:37.691054 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 07:03:37.691151 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:03:37.696717 14588 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:37.698522 14588 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:37.708557 14588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158943df9665432885a51bd6de15662a: Generated new TSK 0
I20260620 07:03:37.709664 14588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:03:37.721382 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14510
I20260620 07:03:37.722611 14572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:37.812609 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14510
2026-06-20T07:03:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1823 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14612](info): set up 2 sockets
Jun 20 07:03:37 dist-test-slave-wvz3 krb5kdc[14612](info): commencing operation
krb5kdc: starting...
W20260620 07:03:38.819408 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:38 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:39.010818 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39641
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:39.338250 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:39.338558 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:39.338686 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:39.346161 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:39.346292 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:39.346354 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:39.346442 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:39.346493 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:39.356904 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:39.358314 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:39.359696 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:39.372059 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:39.372200 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:39.372229 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:39.372836 14625 server_base.cc:1034] running on GCE node
I20260620 07:03:39.373613 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:39.375088 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:39.376332 14625 hybrid_clock.cc:648] HybridClock initialized: now 1781939019376263 us; error 50 us; skew 500 ppm
I20260620 07:03:39.376691 14625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:39 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939019, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:39.383858 14625 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:39.385557 14625 webserver.cc:460] Webserver started at http://127.11.81.254:34115/ using document root <none> and password file <none>
I20260620 07:03:39.386248 14625 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:39.386370 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:39.386669 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:39.389716 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "87b3284f9a994535b5a1f77a57b0d5b8"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "48e7bbc57d769a99dd091c6b27734f89"
server_key_iv: "03d06c9c55e0df15d8e554a8159b3390"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.390470 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "87b3284f9a994535b5a1f77a57b0d5b8"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "48e7bbc57d769a99dd091c6b27734f89"
server_key_iv: "03d06c9c55e0df15d8e554a8159b3390"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.395653 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:39.399147 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:39.400674 14625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:39.400835 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "87b3284f9a994535b5a1f77a57b0d5b8"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "48e7bbc57d769a99dd091c6b27734f89"
server_key_iv: "03d06c9c55e0df15d8e554a8159b3390"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.401013 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:39.418454 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:39.422923 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:39.423205 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:39.441849 14625 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:39641
I20260620 07:03:39.441895 14691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:39641 every 8 connection(s)
I20260620 07:03:39.443238 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:39.447055 14692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:39.449736 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14625
I20260620 07:03:39.449940 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:39.450719 11591 external_mini_cluster.cc:1428] Setting key 62cd91ef575cb0b3f72336410d5965a3
I20260620 07:03:39.460763 14692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8: Bootstrap starting.
I20260620 07:03:39.464321 14692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:39.465305 14692 log.cc:824] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:03:39 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:39.468776 14692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8: No bootstrap required, opened a new log
I20260620 07:03:39.476583 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:39.453152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.11.81.254:39641)
0620 07:03:39.454034 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:39.454051 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:39.454400 (+ 349us) server_negotiation.cc:407] Connection header received
0620 07:03:39.456323 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:39.456354 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:39.456901 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:39.457365 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:39.458594 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:39.460495 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:39.461843 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:39.462888 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:39.463095 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:39.467307 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:39.467319 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:39.467338 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:39.467380 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:39.471477 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:39.472356 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:39.472369 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:39.472378 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:39.472603 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:39.473095 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:39.473100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:39.473104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:39.473844 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:39.474181 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:39.474587 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 07:03:39.475097 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":376,"threads_started":1}
I20260620 07:03:39.479876 14692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } }
I20260620 07:03:39.480350 14692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:39.480453 14692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b3284f9a994535b5a1f77a57b0d5b8, State: Initialized, Role: FOLLOWER
I20260620 07:03:39.481130 14692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } }
I20260620 07:03:39.481383 14692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:39.481571 14692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:39.481721 14692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:39.484630 14692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } }
I20260620 07:03:39.485110 14692 leader_election.cc:302] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b3284f9a994535b5a1f77a57b0d5b8; no voters:
I20260620 07:03:39.485791 14692 leader_election.cc:288] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:39.485884 14697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:39.486685 14697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [term 1 LEADER]: Becoming Leader. State: Replica: 87b3284f9a994535b5a1f77a57b0d5b8, State: Running, Role: LEADER
I20260620 07:03:39.487263 14697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } }
I20260620 07:03:39.487965 14692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:39.489941 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b3284f9a994535b5a1f77a57b0d5b8. Latest consensus state: current_term: 1 leader_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } } }
I20260620 07:03:39.490302 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:39.491252 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b3284f9a994535b5a1f77a57b0d5b8" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39641 } } }
I20260620 07:03:39.491675 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:39.495764 14703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:39.500550 14703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:39.508718 14703 catalog_manager.cc:1347] Generated new cluster ID: fe56ff50f4f84ed1b93a582accc4c45b
I20260620 07:03:39.508855 14703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:39.531953 14703 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:39.533890 14703 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:39.546446 14703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b3284f9a994535b5a1f77a57b0d5b8: Generated new TSK 0
I20260620 07:03:39.547307 14703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:39.604084 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:39.924402 14718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:39.924724 14718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:39.924857 14718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:39.932238 14718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:39.932376 14718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:39.932533 14718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:39.942584 14718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:39.943962 14718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:39.945355 14718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:39.958493 14724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:39.958562 14723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:39.958525 14726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:39.959210 14718 server_base.cc:1034] running on GCE node
I20260620 07:03:39.959843 14718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:39.960789 14718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:39.962041 14718 hybrid_clock.cc:648] HybridClock initialized: now 1781939019961991 us; error 45 us; skew 500 ppm
I20260620 07:03:39.962353 14718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:39 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939019, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:39.965574 14718 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:39.967067 14718 webserver.cc:460] Webserver started at http://127.11.81.193:36139/ using document root <none> and password file <none>
I20260620 07:03:39.967787 14718 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:39.967904 14718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:39.968194 14718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:39.971074 14718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "74276cde2d4341a59c9a173bac00aef9"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "f87a8fe6c90efbec84d3aaec6e9f0856"
server_key_iv: "949c96c2d0c5fa090e5507d1d9eec4c9"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.971876 14718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "74276cde2d4341a59c9a173bac00aef9"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "f87a8fe6c90efbec84d3aaec6e9f0856"
server_key_iv: "949c96c2d0c5fa090e5507d1d9eec4c9"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.977092 14718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:39.980489 14733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:39.981900 14718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:39.982059 14718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "74276cde2d4341a59c9a173bac00aef9"
format_stamp: "Formatted at 2026-06-20 07:03:39 on dist-test-slave-wvz3"
server_key: "f87a8fe6c90efbec84d3aaec6e9f0856"
server_key_iv: "949c96c2d0c5fa090e5507d1d9eec4c9"
server_key_version: "encryptionkey@0"
I20260620 07:03:39.982241 14718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:40.005007 14718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:40.009655 14718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:40.009980 14718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:40.011030 14718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:40.012523 14718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:40.012603 14718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.012727 14718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:40.012778 14718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.044816 14718 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:36789
I20260620 07:03:40.044872 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:36789 every 8 connection(s)
I20260620 07:03:40.046180 14718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:40.054859 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14718
I20260620 07:03:40.055084 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:40.055897 11591 external_mini_cluster.cc:1428] Setting key d250a5cce324d1c6aef980c644b5227c
Jun 20 07:03:40 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939019, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:40.069064 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:40.049125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:49947 (local address 127.11.81.254:39641)
0620 07:03:40.049583 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:40.049589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:40.050717 (+ 1128us) server_negotiation.cc:407] Connection header received
0620 07:03:40.052667 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:40.052672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:40.052747 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:40.052852 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:40.054480 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.055667 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.057503 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.058112 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.058268 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:40.062128 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:40.062136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:40.062142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:40.062188 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:40.065167 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:40.066257 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:40.066262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:40.066265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:40.066413 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:40.067088 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:40.067093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:40.067095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:40.067472 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:40.067600 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:40.068526 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 07:03:40.068716 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260620 07:03:40.070410 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:40.049720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39641 (local address 127.11.81.193:49947)
0620 07:03:40.050535 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:40.050582 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:40.052343 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:40.053088 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:40.053097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:40.053675 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:40.054274 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:40.054295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.055839 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.055844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:40.057310 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:40.057327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.058369 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.058374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:40.058618 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:40.059329 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:40.059365 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:40.061824 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:40.065367 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:40.065377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:40.065397 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:40.066082 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:40.066533 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:40.066538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:40.066541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:40.066802 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:40.067612 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:40.067621 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:40.068080 (+ 459us) client_negotiation.cc:769] Sending connection context
0620 07:03:40.068393 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 07:03:40.068992 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
I20260620 07:03:40.072417 14846 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39641
I20260620 07:03:40.072767 14846 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:40.073570 14846 heartbeater.cc:511] Master 127.11.81.254:39641 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:40.076108 14657 ts_manager.cc:194] Registered new tserver with Master: 74276cde2d4341a59c9a173bac00aef9 (127.11.81.193:36789)
I20260620 07:03:40.077702 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:49947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:40.103961 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:40.426081 14852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:40.426381 14852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:40.426501 14852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:40.434095 14852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:40.434237 14852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:40.434377 14852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:40.444588 14852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:40.445845 14852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:40.447173 14852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:40.459997 14857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:40.459929 14858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:40.460063 14860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:40.460649 14852 server_base.cc:1034] running on GCE node
I20260620 07:03:40.461179 14852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:40.462078 14852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:40.463317 14852 hybrid_clock.cc:648] HybridClock initialized: now 1781939020463231 us; error 41 us; skew 500 ppm
I20260620 07:03:40.463649 14852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:40 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939020, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:40.466742 14852 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:40.468286 14852 webserver.cc:460] Webserver started at http://127.11.81.194:33285/ using document root <none> and password file <none>
I20260620 07:03:40.468951 14852 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:40.469070 14852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:40.469358 14852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:40.472398 14852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "91496ee282ac48988f956764000cb1ee"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "9a56afa975cdecc3b3c0bf67acb26722"
server_key_iv: "445e9e8a797ba14d0ebdd29531ff16cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.473206 14852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "91496ee282ac48988f956764000cb1ee"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "9a56afa975cdecc3b3c0bf67acb26722"
server_key_iv: "445e9e8a797ba14d0ebdd29531ff16cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.478492 14852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:03:40.482244 14867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.483709 14852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:40.483867 14852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "91496ee282ac48988f956764000cb1ee"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "9a56afa975cdecc3b3c0bf67acb26722"
server_key_iv: "445e9e8a797ba14d0ebdd29531ff16cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.484050 14852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:40.502247 14852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:40.506665 14852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:40.506973 14852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:40.508002 14852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:40.509384 14852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:40.509462 14852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.509579 14852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:40.509631 14852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.541715 14852 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:44781
I20260620 07:03:40.541751 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:44781 every 8 connection(s)
I20260620 07:03:40.543040 14852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:40.553404 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14852
I20260620 07:03:40.553587 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:40.554181 11591 external_mini_cluster.cc:1428] Setting key b07c85835fe7c6e999ea954d86984d08
Jun 20 07:03:40 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939020, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:40.563108 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:40.545784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60809 (local address 127.11.81.254:39641)
0620 07:03:40.545996 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:40.546001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:40.547196 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 07:03:40.549023 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:40.549029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:40.549114 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:40.549205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:40.550742 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.551517 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.553076 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.553501 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.553620 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:40.557658 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:40.557664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:40.557667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:40.557700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:40.560178 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:40.560899 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:40.560903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:40.560905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:40.561026 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:40.561459 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:40.561462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:40.561464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:40.561716 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:40.561844 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:40.562545 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 07:03:40.562778 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:03:40.564260 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:40.546203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39641 (local address 127.11.81.194:60809)
0620 07:03:40.547017 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:40.547081 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:40.548738 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:40.549474 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:40.549482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:40.549994 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:40.550562 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:40.550580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.551670 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.551674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:40.552922 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:40.552937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:40.553818 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:40.553826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:40.554155 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:40.554918 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:40.554942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:40.557424 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:40.560323 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:40.560333 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:40.560352 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:40.560766 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:40.561146 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:40.561151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:40.561154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:40.561339 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:40.561843 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:40.561852 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:40.562188 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 07:03:40.562444 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 07:03:40.562983 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":334,"threads_started":1}
I20260620 07:03:40.565694 14980 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39641
I20260620 07:03:40.566047 14980 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:40.566991 14980 heartbeater.cc:511] Master 127.11.81.254:39641 requested a full tablet report, sending...
I20260620 07:03:40.568830 14657 ts_manager.cc:194] Registered new tserver with Master: 91496ee282ac48988f956764000cb1ee (127.11.81.194:44781)
I20260620 07:03:40.569737 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:60809
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:40.596889 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:40.920374 14985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:40.920670 14985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:40.920783 14985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:40.928268 14985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:40.928385 14985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:40.928529 14985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:40.938380 14985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:40.939738 14985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:40.940999 14985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:40.954134 14990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:40.954062 14993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:40.954046 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:40.955013 14985 server_base.cc:1034] running on GCE node
I20260620 07:03:40.955624 14985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:40.956560 14985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:40.957787 14985 hybrid_clock.cc:648] HybridClock initialized: now 1781939020957703 us; error 69 us; skew 500 ppm
I20260620 07:03:40.958103 14985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:40 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939020, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:40.961520 14985 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:40.963060 14985 webserver.cc:460] Webserver started at http://127.11.81.195:41793/ using document root <none> and password file <none>
I20260620 07:03:40.963784 14985 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:40.963907 14985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:40.964187 14985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:40.967104 14985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "b80bcf2ca4f84f25a049fb1c94fc8d73"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "ec4d06097acf431cc5ac7694b95ffb98"
server_key_iv: "ea9142536bdca38e1d46bf5b371b76cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.967969 14985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "b80bcf2ca4f84f25a049fb1c94fc8d73"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "ec4d06097acf431cc5ac7694b95ffb98"
server_key_iv: "ea9142536bdca38e1d46bf5b371b76cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.973408 14985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:03:40.977205 15000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:40.978624 14985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:40.978847 14985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b80bcf2ca4f84f25a049fb1c94fc8d73"
format_stamp: "Formatted at 2026-06-20 07:03:40 on dist-test-slave-wvz3"
server_key: "ec4d06097acf431cc5ac7694b95ffb98"
server_key_iv: "ea9142536bdca38e1d46bf5b371b76cc"
server_key_version: "encryptionkey@0"
I20260620 07:03:40.979048 14985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:40.996302 14985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:41.000691 14985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:41.000986 14985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:41.001992 14985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:41.003440 14985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:41.003521 14985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:41.003650 14985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:41.003706 14985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:41.035650 14985 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33615
I20260620 07:03:41.035697 15112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33615 every 8 connection(s)
I20260620 07:03:41.036909 14985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:41.046859 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 14985
I20260620 07:03:41.047034 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:41.047649 11591 external_mini_cluster.cc:1428] Setting key c6672c2350e56936ef865cbe9375d1b2
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939020, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:41.057658 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.039503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:34157 (local address 127.11.81.254:39641)
0620 07:03:41.039673 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:41.039678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:41.040815 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 07:03:41.042663 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:41.042669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:41.042746 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:41.042835 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:41.044419 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.045552 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.047220 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.047681 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.047824 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.051867 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:41.051872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:41.051875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:41.051909 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:41.054639 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.055419 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.055425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.055429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.055593 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.056089 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.056093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.056095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.056404 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:41.056513 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:41.057220 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 07:03:41.057410 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:03:41.058651 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.039920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39641 (local address 127.11.81.195:34157)
0620 07:03:41.040615 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:41.040657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:41.042354 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:41.043094 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:41.043103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:41.043652 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:41.044217 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:41.044236 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.045706 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.045711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:41.047021 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:41.047037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.048088 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.048092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:41.048372 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.049115 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:41.049144 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:41.051648 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:41.054802 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:41.054809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:41.054823 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:41.055241 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:41.055737 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:41.055740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:41.055743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:41.055952 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:41.056517 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:41.056524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:41.056881 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 07:03:41.057108 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 07:03:41.057533 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260620 07:03:41.060071 15113 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39641
I20260620 07:03:41.060374 15113 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:41.061124 15113 heartbeater.cc:511] Master 127.11.81.254:39641 requested a full tablet report, sending...
I20260620 07:03:41.062691 14657 ts_manager.cc:194] Registered new tserver with Master: b80bcf2ca4f84f25a049fb1c94fc8d73 (127.11.81.195:33615)
I20260620 07:03:41.063413 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:41.063740 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:34157
I20260620 07:03:41.081185 14846 heartbeater.cc:503] Master 127.11.81.254:39641 was elected leader, sending a full tablet report...
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939021, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939021, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:41.109725 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.096113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.11.81.254:39641)
0620 07:03:41.096340 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:41.096347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:41.096709 (+ 362us) server_negotiation.cc:407] Connection header received
0620 07:03:41.096949 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:41.096953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:41.097017 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:41.097099 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:41.098403 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.099103 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.100373 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.100787 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.100897 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.104737 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:41.104743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:41.104746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:41.104788 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:41.107152 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.107807 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.107813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.107817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.107971 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.108355 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.108360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.108363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.108668 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:41.108910 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:41.109209 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 07:03:41.109430 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 07:03:41.110273 14657 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45652
I20260620 07:03:41.112214 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14718
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939021, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:41.128499 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.113327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.11.81.193:36789)
0620 07:03:41.113955 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:41.113962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:41.113986 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:03:41.114041 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:41.114048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:41.114260 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:41.114391 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:41.115653 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.116752 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.118026 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.118596 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.118699 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.122833 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:41.122847 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:41.122864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:41.122902 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:41.125776 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.126482 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.126490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.126492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.126626 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.126928 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.126931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.126933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.127274 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:41.127544 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:41.127910 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 07:03:41.128142 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":315,"threads_started":1}
W20260620 07:03:41.129855 14841 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40186
W20260620 07:03:41.131006 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:41.131232 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14718
I20260620 07:03:41.144001 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14852
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939021, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:41.160445 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.144926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.11.81.194:44781)
0620 07:03:41.145504 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:41.145510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:41.145536 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:41.145598 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:41.145605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:41.145884 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:41.146019 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:41.147225 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.148311 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.149611 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.150186 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.150304 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.154633 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:41.154645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:41.154660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:41.154699 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:41.157585 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.158314 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.158321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.158324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.158454 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.158828 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.158832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.158834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.159185 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:41.159503 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:41.159832 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 07:03:41.160098 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":333,"threads_started":1}
W20260620 07:03:41.161722 14975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53946
W20260620 07:03:41.162881 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14852: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:41.163018 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14852
I20260620 07:03:41.175199 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14985
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[14612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939021, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:41.192694 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:41.176113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.11.81.195:33615)
0620 07:03:41.176686 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:41.176692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:41.176722 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:03:41.176790 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:41.176796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:41.177009 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:41.177135 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:41.178570 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.179803 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.181260 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:41.181857 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:41.181961 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:41.186408 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:41.186422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:41.186437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:41.186480 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:41.189664 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.190466 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.190475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.190477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.190660 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:41.191048 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:41.191052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:41.191055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:41.191569 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:41.191870 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:41.192070 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 07:03:41.192335 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":264,"threads_started":1}
W20260620 07:03:41.194079 15108 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47248
W20260620 07:03:41.195217 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:03:41.195415 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14985
I20260620 07:03:41.207458 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 14625
I20260620 07:03:41.208739 14687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:41.310990 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 14625
2026-06-20T07:03:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3504 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[15130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[15130](info): set up 2 sockets
Jun 20 07:03:41 dist-test-slave-wvz3 krb5kdc[15130](info): commencing operation
krb5kdc: starting...
W20260620 07:03:42.323515 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:42 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:42.514644 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36031
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:42.832773 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:42.833081 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:42.833189 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:42.840550 15143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:42.840674 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:42.840719 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:42.840780 15143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:42.840842 15143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:42.850775 15143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:36031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:42.852167 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:42.853470 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:42.865253 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:42.865401 15149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:42.865679 15151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:42.866155 15143 server_base.cc:1034] running on GCE node
I20260620 07:03:42.866910 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:42.868440 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:42.869668 15143 hybrid_clock.cc:648] HybridClock initialized: now 1781939022869620 us; error 35 us; skew 500 ppm
I20260620 07:03:42.870019 15143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:42 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939022, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:42.873270 15143 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:42.874990 15143 webserver.cc:460] Webserver started at http://127.11.81.254:42021/ using document root <none> and password file <none>
I20260620 07:03:42.875767 15143 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:42.875876 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:42.876195 15143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:42.879262 15143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "c48ab7f6a6c54b318698a6a88742b5a4"
format_stamp: "Formatted at 2026-06-20 07:03:42 on dist-test-slave-wvz3"
server_key: "f61232f51ee207395c05bc844452777e"
server_key_iv: "4e58ad95a2b95453f0ecbf5d86ae177a"
server_key_version: "encryptionkey@0"
I20260620 07:03:42.880177 15143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "c48ab7f6a6c54b318698a6a88742b5a4"
format_stamp: "Formatted at 2026-06-20 07:03:42 on dist-test-slave-wvz3"
server_key: "f61232f51ee207395c05bc844452777e"
server_key_iv: "4e58ad95a2b95453f0ecbf5d86ae177a"
server_key_version: "encryptionkey@0"
I20260620 07:03:42.886159 15143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 07:03:42.889878 15158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:42.891449 15143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:03:42.891644 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c48ab7f6a6c54b318698a6a88742b5a4"
format_stamp: "Formatted at 2026-06-20 07:03:42 on dist-test-slave-wvz3"
server_key: "f61232f51ee207395c05bc844452777e"
server_key_iv: "4e58ad95a2b95453f0ecbf5d86ae177a"
server_key_version: "encryptionkey@0"
I20260620 07:03:42.891844 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:42.909576 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:42.914367 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:42.914657 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:42.933234 15143 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:36031
I20260620 07:03:42.933235 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:36031 every 8 connection(s)
I20260620 07:03:42.934688 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:42.938573 15210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:42.943758 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15143
I20260620 07:03:42.944017 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:42.944787 11591 external_mini_cluster.cc:1428] Setting key dc3818df34c82d13762f96ae6e785d54
I20260620 07:03:42.952070 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4: Bootstrap starting.
I20260620 07:03:42.955951 15210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:42.957259 15210 log.cc:824] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:42.960130 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4: No bootstrap required, opened a new log
Jun 20 07:03:42 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:42.968318 15210 raft_consensus.cc:348] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } }
I20260620 07:03:42.968698 15210 raft_consensus.cc:374] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:42.968798 15210 raft_consensus.cc:729] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48ab7f6a6c54b318698a6a88742b5a4, State: Initialized, Role: FOLLOWER
I20260620 07:03:42.969422 15210 consensus_queue.cc:260] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } }
I20260620 07:03:42.969660 15210 raft_consensus.cc:388] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:42.969818 15210 raft_consensus.cc:482] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:42.970036 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:42.971181 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:42.947032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.11.81.254:36031)
0620 07:03:42.950263 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:42.950281 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:42.950334 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:03:42.952691 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:42.952721 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:42.953245 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:42.953707 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:42.955125 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:42.957084 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:42.958425 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:42.959159 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:42.959361 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:42.963587 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:42.963601 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:42.963619 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:42.963662 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:42.966840 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:42.967696 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:42.967703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:42.967711 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:42.967886 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:42.968285 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:42.968289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:42.968291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:42.968952 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:42.969242 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:42.969672 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 07:03:42.970097 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2979,"thread_start_us":2739,"threads_started":1}
I20260620 07:03:42.972385 15210 raft_consensus.cc:504] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } }
I20260620 07:03:42.972858 15210 leader_election.cc:302] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c48ab7f6a6c54b318698a6a88742b5a4; no voters:
I20260620 07:03:42.973426 15210 leader_election.cc:288] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:42.973567 15215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:42.974905 15215 raft_consensus.cc:686] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [term 1 LEADER]: Becoming Leader. State: Replica: c48ab7f6a6c54b318698a6a88742b5a4, State: Running, Role: LEADER
I20260620 07:03:42.975442 15215 consensus_queue.cc:237] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } }
I20260620 07:03:42.975586 15210 sys_catalog.cc:564] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:42.978608 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c48ab7f6a6c54b318698a6a88742b5a4. Latest consensus state: current_term: 1 leader_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } } }
I20260620 07:03:42.978902 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:42.979455 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48ab7f6a6c54b318698a6a88742b5a4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36031 } } }
I20260620 07:03:42.980015 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:42.980211 15221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:42.985942 15221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:42.995748 15221 catalog_manager.cc:1347] Generated new cluster ID: 7d6eb22631284727ad1dabb199ef113e
I20260620 07:03:42.995882 15221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:43.017827 15221 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:43.019449 15221 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:43.031837 15221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c48ab7f6a6c54b318698a6a88742b5a4: Generated new TSK 0
I20260620 07:03:43.032828 15221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:43.099006 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:43.444347 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:43.444654 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:43.444762 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:43.452778 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:43.452934 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:43.453114 15236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:43.464076 15236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:43.465432 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:43.466795 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:43.482709 15244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:43.482877 15242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:43.483017 15241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:43.483542 15236 server_base.cc:1034] running on GCE node
I20260620 07:03:43.485201 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:43.486765 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:43.488070 15236 hybrid_clock.cc:648] HybridClock initialized: now 1781939023487996 us; error 60 us; skew 500 ppm
I20260620 07:03:43.488457 15236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:43 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939023, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:43.491918 15236 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:43.493440 15236 webserver.cc:460] Webserver started at http://127.11.81.193:45169/ using document root <none> and password file <none>
I20260620 07:03:43.494258 15236 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:43.494385 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:43.494717 15236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:43.497815 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "5ffd90319e694974824f0e286c1c5a18"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "e1830b81415ca0c44ad06c00b5319567"
server_key_iv: "8fb2a447dd5596362fbb90ba73a13902"
server_key_version: "encryptionkey@0"
I20260620 07:03:43.498591 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "5ffd90319e694974824f0e286c1c5a18"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "e1830b81415ca0c44ad06c00b5319567"
server_key_iv: "8fb2a447dd5596362fbb90ba73a13902"
server_key_version: "encryptionkey@0"
I20260620 07:03:43.503770 15236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 07:03:43.507252 15251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:43.508785 15236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:43.508953 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "5ffd90319e694974824f0e286c1c5a18"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "e1830b81415ca0c44ad06c00b5319567"
server_key_iv: "8fb2a447dd5596362fbb90ba73a13902"
server_key_version: "encryptionkey@0"
I20260620 07:03:43.509157 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:43.523988 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:43.528707 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:43.528983 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:43.530026 15236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:43.531498 15236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:43.531587 15236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:43.531703 15236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:43.531759 15236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:43.562350 15236 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:35555
I20260620 07:03:43.562387 15363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:35555 every 8 connection(s)
I20260620 07:03:43.563700 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:43.570921 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15236
I20260620 07:03:43.571127 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:43.571770 11591 external_mini_cluster.cc:1428] Setting key cba921ab6b768aee60fa462a9f1bbf4d
Jun 20 07:03:43 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939023, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:43.583853 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:43.566127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:44477 (local address 127.11.81.254:36031)
0620 07:03:43.566528 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:43.566540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:43.567466 (+ 926us) server_negotiation.cc:407] Connection header received
0620 07:03:43.569320 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:43.569326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:43.569415 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:43.569509 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:43.571019 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:43.571843 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:43.573784 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:43.574210 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:43.574321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:43.577959 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:43.577966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:43.577970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:43.578006 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:43.580465 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:43.581169 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:43.581175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:43.581177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:43.581294 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:43.581760 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:43.581765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:43.581768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:43.582169 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:43.582308 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:43.583269 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 07:03:43.583511 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":211,"threads_started":1}
I20260620 07:03:43.585224 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:43.566625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36031 (local address 127.11.81.193:44477)
0620 07:03:43.567310 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:43.567354 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:43.569049 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:43.569751 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:43.569759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:43.570257 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:43.570853 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:43.570870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:43.572005 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:43.572011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:43.573649 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:43.573662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:43.574428 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:43.574435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:43.574671 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:43.575372 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:43.575395 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:43.577744 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:43.580602 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:43.580612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:43.580631 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:43.581050 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:43.581403 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:43.581408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:43.581412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:43.581646 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:43.582317 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:43.582326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:43.582766 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 07:03:43.583175 (+ 409us) client_negotiation.cc:240] Negotiation successful
0620 07:03:43.583744 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":249,"threads_started":1}
I20260620 07:03:43.587064 15364 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36031
I20260620 07:03:43.587461 15364 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:43.588445 15364 heartbeater.cc:511] Master 127.11.81.254:36031 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:43.591061 15175 ts_manager.cc:194] Registered new tserver with Master: 5ffd90319e694974824f0e286c1c5a18 (127.11.81.193:35555)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:03:43.592676 15175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:44477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:43.616986 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:43.948287 15370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:43.948577 15370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:43.948684 15370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:43.956364 15370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:43.956493 15370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:43.956615 15370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:43.969282 15370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:43.970552 15370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:43.971997 15370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:43.985931 15375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:43.985905 15376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:43.986140 15378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:43.987040 15370 server_base.cc:1034] running on GCE node
I20260620 07:03:43.987746 15370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:43.988715 15370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:43.989953 15370 hybrid_clock.cc:648] HybridClock initialized: now 1781939023989899 us; error 63 us; skew 500 ppm
I20260620 07:03:43.990298 15370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:43 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939023, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:43.993479 15370 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:43.995191 15370 webserver.cc:460] Webserver started at http://127.11.81.194:39769/ using document root <none> and password file <none>
I20260620 07:03:43.995955 15370 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:43.996068 15370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:43.996371 15370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:43.999275 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "c80e4b2ad0274a9084be56b967a1feea"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "b608de37c5d84b2e8ab4c5678577b99e"
server_key_iv: "25a6921d6b2d3427cdde71eeb017c2e5"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.000101 15370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "c80e4b2ad0274a9084be56b967a1feea"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "b608de37c5d84b2e8ab4c5678577b99e"
server_key_iv: "25a6921d6b2d3427cdde71eeb017c2e5"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.005465 15370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:44.009224 15385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.010710 15370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:03:44.010879 15370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "c80e4b2ad0274a9084be56b967a1feea"
format_stamp: "Formatted at 2026-06-20 07:03:43 on dist-test-slave-wvz3"
server_key: "b608de37c5d84b2e8ab4c5678577b99e"
server_key_iv: "25a6921d6b2d3427cdde71eeb017c2e5"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.011083 15370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:44.031596 15370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:44.036240 15370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:44.036527 15370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:44.037519 15370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:44.039007 15370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:44.039086 15370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.039227 15370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:44.039278 15370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.070252 15370 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:43139
I20260620 07:03:44.070297 15497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:43139 every 8 connection(s)
I20260620 07:03:44.071538 15370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:44.076567 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15370
I20260620 07:03:44.076756 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:44.077262 11591 external_mini_cluster.cc:1428] Setting key 9c22f41deff26104a09eef4daf5d93b4
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939023, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:44.094978 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.074098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:45695 (local address 127.11.81.254:36031)
0620 07:03:44.074299 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.074306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.075669 (+ 1363us) server_negotiation.cc:407] Connection header received
0620 07:03:44.078011 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.078018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.078108 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.078191 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.080149 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.080979 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.083582 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.084078 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.084184 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.088268 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.088276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.088282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.088323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.091438 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.092260 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.092265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.092269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.092426 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.092969 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.092974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.092977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.093358 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.093513 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.094369 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.094625 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:03:44.096226 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.074696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36031 (local address 127.11.81.194:45695)
0620 07:03:44.075506 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:44.075550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:44.077662 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:44.078498 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:44.078507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:44.079115 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:44.079961 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:44.079988 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.081197 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.081204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:44.083260 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:44.083387 (+ 127us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.084319 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.084325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:44.084619 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.085365 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:44.085403 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:44.087974 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:44.091622 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:44.091631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:44.091646 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:44.092118 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:44.092574 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:44.092579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:44.092581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:44.092823 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:44.093508 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:44.093515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:44.093899 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 07:03:44.094233 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 07:03:44.094691 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:03:44.097723 15498 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36031
I20260620 07:03:44.098066 15498 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:44.098783 15498 heartbeater.cc:511] Master 127.11.81.254:36031 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
I20260620 07:03:44.100497 15175 ts_manager.cc:194] Registered new tserver with Master: c80e4b2ad0274a9084be56b967a1feea (127.11.81.194:43139)
I20260620 07:03:44.101351 15175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:45695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:44.122305 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:44.439633 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:44.439904 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:44.440025 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:44.447156 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:44.447273 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:44.447445 15503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:44.457168 15503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:44.458446 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:44.459844 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:44.472714 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:44.472812 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:44.472788 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:44.474239 15503 server_base.cc:1034] running on GCE node
I20260620 07:03:44.474879 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:44.476081 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:44.477319 15503 hybrid_clock.cc:648] HybridClock initialized: now 1781939024477225 us; error 83 us; skew 500 ppm
I20260620 07:03:44.477660 15503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:44.484020 15503 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:44.485621 15503 webserver.cc:460] Webserver started at http://127.11.81.195:35969/ using document root <none> and password file <none>
I20260620 07:03:44.486353 15503 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:44.486459 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:44.486716 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:44.489903 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "ebfd9d30d55e4209ab9e1eae54642f67"
format_stamp: "Formatted at 2026-06-20 07:03:44 on dist-test-slave-wvz3"
server_key: "484d17aadee081652ce7171a9688f185"
server_key_iv: "87110eec3d7cf99d9b52b764c021c07e"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.490659 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "ebfd9d30d55e4209ab9e1eae54642f67"
format_stamp: "Formatted at 2026-06-20 07:03:44 on dist-test-slave-wvz3"
server_key: "484d17aadee081652ce7171a9688f185"
server_key_iv: "87110eec3d7cf99d9b52b764c021c07e"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.496013 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:03:44.499589 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.501013 15503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:44.501161 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "ebfd9d30d55e4209ab9e1eae54642f67"
format_stamp: "Formatted at 2026-06-20 07:03:44 on dist-test-slave-wvz3"
server_key: "484d17aadee081652ce7171a9688f185"
server_key_iv: "87110eec3d7cf99d9b52b764c021c07e"
server_key_version: "encryptionkey@0"
I20260620 07:03:44.501387 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:44.519438 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:44.523701 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:44.523988 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:44.525035 15503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:44.526418 15503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:44.526494 15503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.526608 15503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:44.526670 15503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:44.557015 15503 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42189
I20260620 07:03:44.557061 15630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42189 every 8 connection(s)
I20260620 07:03:44.558259 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:44.561229 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15503
I20260620 07:03:44.561434 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:44.561966 11591 external_mini_cluster.cc:1428] Setting key 62673d80f4caab4f06cd3d30bca2dbaf
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:44.579107 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.560675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:52305 (local address 127.11.81.254:36031)
0620 07:03:44.560857 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.560862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.562344 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 07:03:44.564908 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.564913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.564981 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.565059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.567029 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.567867 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.569348 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.569733 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.569823 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.573121 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.573126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.573129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.573160 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.575986 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.576969 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.576973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.576975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.577107 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.577628 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.577631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.577634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.577898 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.578018 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.578683 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.578848 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:03:44.580093 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.561225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36031 (local address 127.11.81.195:52305)
0620 07:03:44.562178 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:44.562224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:44.564604 (+ 2380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:44.565339 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:44.565352 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:44.566103 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:44.566849 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:44.566874 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.568023 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.568027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:44.569228 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:44.569240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.569909 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.569914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:44.570104 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.570736 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:44.570762 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:44.572917 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:44.576130 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:44.576140 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:44.576160 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:44.576808 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:44.577230 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:44.577235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:44.577238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:44.577523 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:44.578004 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:44.578010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:44.578340 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 07:03:44.578614 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 07:03:44.578996 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":340,"threads_started":1}
I20260620 07:03:44.581588 15631 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36031
I20260620 07:03:44.581866 15631 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:44.582662 15631 heartbeater.cc:511] Master 127.11.81.254:36031 requested a full tablet report, sending...
I20260620 07:03:44.584151 15175 ts_manager.cc:194] Registered new tserver with Master: ebfd9d30d55e4209ab9e1eae54642f67 (127.11.81.195:42189)
I20260620 07:03:44.584960 15175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:52305
I20260620 07:03:44.589659 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:44.595940 15364 heartbeater.cc:503] Master 127.11.81.254:36031 was elected leader, sending a full tablet report...
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:44.637408 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.622341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.11.81.193:35555)
0620 07:03:44.622942 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.622949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.622977 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:03:44.623086 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.623092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.623361 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.623549 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.625038 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.626064 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.627354 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.627939 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.628038 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.631886 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.631900 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.631915 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.631958 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.634713 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.635452 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.635459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.635462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.635591 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.635961 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.635964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.635966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.636321 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.636590 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.636816 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.637018 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":302,"threads_started":1}
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:44.654577 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.641543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.11.81.254:36031)
0620 07:03:44.641744 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.641750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.641938 (+ 188us) server_negotiation.cc:407] Connection header received
0620 07:03:44.642109 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.642113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.642175 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.642255 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.643600 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.644306 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.645505 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.645917 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.646033 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.649758 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.649765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.649770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.649806 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.652154 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.652772 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.652777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.652779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.652926 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.653249 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.653256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.653259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.653634 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.653866 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.654091 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.654309 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 07:03:44.655231 15175 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43982
I20260620 07:03:44.656277 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15236
I20260620 07:03:44.668952 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.657153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.11.81.193:35555)
0620 07:03:44.657359 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.657366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.657806 (+ 440us) server_negotiation.cc:407] Connection header received
0620 07:03:44.657916 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.657920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.657981 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.658056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.659199 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.660030 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.661184 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.661604 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.661704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.663933 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.663938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.663942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.663973 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.666428 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.667124 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.667130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.667134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.667262 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.667648 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.667654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.667657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.667920 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.668138 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.668431 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.668647 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:03:44.669538 15359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:44.751376 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15236
I20260620 07:03:44.764022 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15370
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:44.779260 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.764915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.11.81.194:43139)
0620 07:03:44.765422 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.765429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.765454 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:44.765513 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.765519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.765733 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.765868 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.766943 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.768010 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.769215 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.769778 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.769880 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.773779 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.773790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.773807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.773841 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.776735 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.777429 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.777437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.777439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.777569 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.777888 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.777892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.777894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.778230 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.778493 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.778709 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.778910 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260620 07:03:44.780663 15493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:44.862300 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15370
I20260620 07:03:44.875046 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15503
Jun 20 07:03:44 dist-test-slave-wvz3 krb5kdc[15130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:44.891485 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:44.876006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.11.81.195:42189)
0620 07:03:44.876626 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:44.876633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:44.876662 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:03:44.876722 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:44.876728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:44.876948 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:44.877091 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:44.878336 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.879427 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.880794 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:44.881405 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:44.881526 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:44.885906 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:44.885920 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:44.885935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:44.885973 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:44.888650 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.889358 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.889366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.889368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.889493 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:44.889862 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:44.889866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:44.889868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:44.890211 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:44.890512 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:44.890832 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 07:03:44.891094 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260620 07:03:44.892751 15626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:44.974433 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15503
I20260620 07:03:44.987084 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15143
I20260620 07:03:44.988345 15205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:45.092411 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15143
2026-06-20T07:03:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3780 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:45 dist-test-slave-wvz3 krb5kdc[15650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:45 dist-test-slave-wvz3 krb5kdc[15650](info): set up 2 sockets
Jun 20 07:03:45 dist-test-slave-wvz3 krb5kdc[15650](info): commencing operation
krb5kdc: starting...
W20260620 07:03:46.099586 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:46 dist-test-slave-wvz3 krb5kdc[15650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:46.295913 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35185
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:46.614181 15663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:46.614502 15663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:46.614634 15663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:46.621928 15663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:46.622078 15663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:46.622167 15663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:46.622300 15663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:46.622402 15663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:46.633392 15663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:46.634756 15663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:46.636092 15663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:46.648204 15668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:46.648204 15671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:46.648238 15669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:46.649421 15663 server_base.cc:1034] running on GCE node
I20260620 07:03:46.650231 15663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:46.651994 15663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:46.653234 15663 hybrid_clock.cc:648] HybridClock initialized: now 1781939026653169 us; error 57 us; skew 500 ppm
I20260620 07:03:46.653561 15663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:46 dist-test-slave-wvz3 krb5kdc[15650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:46.656661 15663 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:46.657784 15663 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:46.657873 15663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:46.658140 15663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:46.661091 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "93a4345e7e524869b765c593977b7e8d"
format_stamp: "Formatted at 2026-06-20 07:03:46 on dist-test-slave-wvz3"
server_key: "9e132971f54ab06e26cffa5fdb762d89"
server_key_iv: "cd17ae7770735993a9fe098e0f535206"
server_key_version: "encryptionkey@0"
I20260620 07:03:46.661881 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "93a4345e7e524869b765c593977b7e8d"
format_stamp: "Formatted at 2026-06-20 07:03:46 on dist-test-slave-wvz3"
server_key: "9e132971f54ab06e26cffa5fdb762d89"
server_key_iv: "cd17ae7770735993a9fe098e0f535206"
server_key_version: "encryptionkey@0"
I20260620 07:03:46.667012 15663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:46.670600 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:46.672199 15663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:46.672394 15663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "93a4345e7e524869b765c593977b7e8d"
format_stamp: "Formatted at 2026-06-20 07:03:46 on dist-test-slave-wvz3"
server_key: "9e132971f54ab06e26cffa5fdb762d89"
server_key_iv: "cd17ae7770735993a9fe098e0f535206"
server_key_version: "encryptionkey@0"
I20260620 07:03:46.672641 15663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:46.689921 15663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:46.694756 15663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:46.695076 15663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:46.713955 15663 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35185
I20260620 07:03:46.713930 15728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35185 every 8 connection(s)
I20260620 07:03:46.714917 15663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:46.718534 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:46.724720 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15663
I20260620 07:03:46.724927 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:46.725529 11591 external_mini_cluster.cc:1428] Setting key b439035bdf609a440ce5d075f15c07a3
I20260620 07:03:46.729379 15729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: Bootstrap starting.
I20260620 07:03:46.733222 15729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:46.734587 15729 log.cc:824] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:46.737903 15729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: No bootstrap required, opened a new log
Jun 20 07:03:46 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:46.746980 15729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } }
I20260620 07:03:46.747502 15729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:46.747617 15729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a4345e7e524869b765c593977b7e8d, State: Initialized, Role: FOLLOWER
I20260620 07:03:46.748361 15729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } }
I20260620 07:03:46.748641 15729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:46.748783 15729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:46.748970 15729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:46.750687 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:46.727559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.11.81.254:35185)
0620 07:03:46.728397 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:46.728413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:46.728461 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:03:46.730838 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:46.730869 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:46.731459 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:46.731943 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:46.733406 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:46.735411 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:46.736864 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:46.737865 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:46.738048 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:46.741929 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:46.741940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:46.741961 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:46.742009 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:46.746073 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:46.746858 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:46.746867 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:46.746877 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:46.747097 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:46.747565 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:46.747570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:46.747574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:46.748327 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:46.748616 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:46.749117 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 07:03:46.749557 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":335,"threads_started":1}
I20260620 07:03:46.751504 15729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } }
I20260620 07:03:46.751978 15729 leader_election.cc:302] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a4345e7e524869b765c593977b7e8d; no voters:
I20260620 07:03:46.752607 15729 leader_election.cc:288] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:46.752732 15734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:46.753700 15734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [term 1 LEADER]: Becoming Leader. State: Replica: 93a4345e7e524869b765c593977b7e8d, State: Running, Role: LEADER
I20260620 07:03:46.754302 15734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } }
I20260620 07:03:46.754845 15729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:46.757339 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93a4345e7e524869b765c593977b7e8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } } }
I20260620 07:03:46.757738 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:46.762848 15736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a4345e7e524869b765c593977b7e8d. Latest consensus state: current_term: 1 leader_uuid: "93a4345e7e524869b765c593977b7e8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a4345e7e524869b765c593977b7e8d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35185 } } }
I20260620 07:03:46.763248 15736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:46.763608 15743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:46.769577 15743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:46.779191 15743 catalog_manager.cc:1347] Generated new cluster ID: 56b8e3c1697b4f6c86071c33efe81e96
I20260620 07:03:46.779335 15743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:46.802278 15743 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:46.803686 15743 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:46.811918 15743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: Generated new TSK 0
I20260620 07:03:46.812799 15743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:46.870903 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:47.200634 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:47.200927 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:47.201066 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:47.208474 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:47.208601 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:47.208752 15755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:47.218652 15755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:47.220036 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:47.221314 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:47.233843 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:47.234067 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:47.233814 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:47.234750 15755 server_base.cc:1034] running on GCE node
I20260620 07:03:47.235404 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:47.236409 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:47.237656 15755 hybrid_clock.cc:648] HybridClock initialized: now 1781939027237600 us; error 52 us; skew 500 ppm
I20260620 07:03:47.237973 15755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:47 dist-test-slave-wvz3 krb5kdc[15650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939027, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:47.241344 15755 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:47.242601 15755 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:47.242697 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:47.243022 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:47.246119 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "5595b749597240e98756fb25a9a60701"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "ec6ceb33f0fa75e7e8d467715fa64513"
server_key_iv: "c8aea0886ab6456c57141d862085c12e"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.246944 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "5595b749597240e98756fb25a9a60701"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "ec6ceb33f0fa75e7e8d467715fa64513"
server_key_iv: "c8aea0886ab6456c57141d862085c12e"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.252445 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:03:47.256219 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.257737 15755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:47.257932 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "5595b749597240e98756fb25a9a60701"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "ec6ceb33f0fa75e7e8d467715fa64513"
server_key_iv: "c8aea0886ab6456c57141d862085c12e"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.258200 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:47.279263 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:47.283428 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:47.283689 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:47.284663 15755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:47.286077 15755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:47.286190 15755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.286298 15755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:47.286374 15755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.316512 15755 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:43669
I20260620 07:03:47.316545 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:43669 every 8 connection(s)
I20260620 07:03:47.317354 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:47.322461 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15755
I20260620 07:03:47.322651 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:47.323391 11591 external_mini_cluster.cc:1428] Setting key c646c119dad05fcdc2fe4d5b758c6f39
Jun 20 07:03:47 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939027, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:47.337818 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:47.319736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:38575 (local address 127.11.81.254:35185)
0620 07:03:47.320159 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:47.320163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:47.321119 (+ 956us) server_negotiation.cc:407] Connection header received
0620 07:03:47.322987 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:47.322994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:47.323081 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:47.323208 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:47.324815 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.325745 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.327384 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.327912 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.328047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:47.331745 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:47.331751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:47.331754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:47.331789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:47.334580 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:47.335407 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:47.335412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:47.335414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:47.335542 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:47.336001 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:47.336005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:47.336007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:47.336292 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:47.336438 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:47.337217 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 07:03:47.337449 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":203,"threads_started":1}
I20260620 07:03:47.338837 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:47.320302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35185 (local address 127.11.81.193:38575)
0620 07:03:47.320951 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:47.320994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:47.322712 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:47.323429 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:47.323438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:47.324030 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:47.324658 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:47.324676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.325908 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.325912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:47.327208 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:47.327221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.328150 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.328156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:47.328349 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:47.329011 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:47.329033 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:47.331486 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:47.334757 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:47.334769 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:47.334786 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:47.335251 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:47.335665 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:47.335670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:47.335674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:47.335864 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:47.336433 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:47.336441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:47.336815 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 07:03:47.337070 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 07:03:47.337654 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:47.340458 15882 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35185
I20260620 07:03:47.340740 15882 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:47.341413 15882 heartbeater.cc:511] Master 127.11.81.254:35185 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:03:47.343734 15694 ts_manager.cc:194] Registered new tserver with Master: 5595b749597240e98756fb25a9a60701 (127.11.81.193:43669)
I20260620 07:03:47.345172 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:38575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:47.365921 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:47.694842 15888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:47.695143 15888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:47.695273 15888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:47.702689 15888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:47.702804 15888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:47.702970 15888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:47.712860 15888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:47.714216 15888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:47.715534 15888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:47.729388 15893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:47.729318 15894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:47.729350 15896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:47.732016 15888 server_base.cc:1034] running on GCE node
I20260620 07:03:47.732637 15888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:47.733541 15888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:47.734773 15888 hybrid_clock.cc:648] HybridClock initialized: now 1781939027734678 us; error 90 us; skew 500 ppm
I20260620 07:03:47.735126 15888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:47 dist-test-slave-wvz3 krb5kdc[15650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939027, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:47.738219 15888 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:47.739423 15888 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:47.739529 15888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:47.739862 15888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:47.742813 15888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "ff3828edbf3446598a12a73eb047fd6b"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "cb0c180755cc86fe002eed463cc616b1"
server_key_iv: "6804967c06c607ee2aa1395af120339c"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.743674 15888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "ff3828edbf3446598a12a73eb047fd6b"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "cb0c180755cc86fe002eed463cc616b1"
server_key_iv: "6804967c06c607ee2aa1395af120339c"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.748769 15888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:47.752286 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.753786 15888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:03:47.753960 15888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "ff3828edbf3446598a12a73eb047fd6b"
format_stamp: "Formatted at 2026-06-20 07:03:47 on dist-test-slave-wvz3"
server_key: "cb0c180755cc86fe002eed463cc616b1"
server_key_iv: "6804967c06c607ee2aa1395af120339c"
server_key_version: "encryptionkey@0"
I20260620 07:03:47.754168 15888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:47.772333 15888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:47.776598 15888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:47.776849 15888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:47.777818 15888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:47.779140 15888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:47.779240 15888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.779378 15888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:47.779448 15888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:47.809790 15888 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:35931
I20260620 07:03:47.809836 16014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:35931 every 8 connection(s)
I20260620 07:03:47.810597 15888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:47.814522 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 15888
I20260620 07:03:47.814702 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:47.815124 11591 external_mini_cluster.cc:1428] Setting key e126322d7fe6acd42a04c76c16ec3c9b
Jun 20 07:03:47 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939027, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:47.830681 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:47.812920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60771 (local address 127.11.81.254:35185)
0620 07:03:47.813123 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:47.813127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:47.814334 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 07:03:47.816409 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:47.816414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:47.816476 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:47.816576 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:47.818316 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.819057 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.820644 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.821069 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.821179 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:47.824966 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:47.824971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:47.824974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:47.825010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:47.827339 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:47.828223 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:47.828228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:47.828230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:47.828363 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:47.828868 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:47.828872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:47.828874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:47.829225 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:47.829367 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:47.830100 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 07:03:47.830331 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:03:47.831769 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:47.813406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35185 (local address 127.11.81.194:60771)
0620 07:03:47.814172 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:47.814210 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:47.816102 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:47.816856 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:47.816868 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:47.817491 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:47.818128 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:47.818153 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.819203 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.819207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:47.820504 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:47.820516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:47.821286 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:47.821293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:47.821515 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:47.822180 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:47.822204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:47.824742 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:47.827472 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:47.827481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:47.827497 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:47.828069 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:47.828470 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:47.828475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:47.828478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:47.828722 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:47.829376 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:47.829385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:47.829736 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 07:03:47.829970 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 07:03:47.830566 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:03:47.833246 16015 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35185
I20260620 07:03:47.833526 16015 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:47.834161 16015 heartbeater.cc:511] Master 127.11.81.254:35185 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
I20260620 07:03:47.835637 15694 ts_manager.cc:194] Registered new tserver with Master: ff3828edbf3446598a12a73eb047fd6b (127.11.81.194:35931)
I20260620 07:03:47.836454 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:60771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:47.858465 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:48.171110 16020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:48.171442 16020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:48.171554 16020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:48.178740 16020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:48.178846 16020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:48.178968 16020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:48.188851 16020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:48.190150 16020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:48.191462 16020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:48.204327 16025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:48.204258 16026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:48.204299 16028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:48.205224 16020 server_base.cc:1034] running on GCE node
I20260620 07:03:48.205812 16020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:48.206832 16020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:48.208081 16020 hybrid_clock.cc:648] HybridClock initialized: now 1781939028208030 us; error 41 us; skew 500 ppm
I20260620 07:03:48.208405 16020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:48 dist-test-slave-wvz3 krb5kdc[15650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939028, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:48.211692 16020 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:03:48.213025 16020 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:48.213132 16020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:48.213455 16020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:48.216526 16020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "18f716a7d81048f89ff4fbf2e8ef5010"
format_stamp: "Formatted at 2026-06-20 07:03:48 on dist-test-slave-wvz3"
server_key: "7444c208f70e40da16bf1162b9c74c13"
server_key_iv: "fdad3b26e4cb378283ececc88404cf25"
server_key_version: "encryptionkey@0"
I20260620 07:03:48.217327 16020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "18f716a7d81048f89ff4fbf2e8ef5010"
format_stamp: "Formatted at 2026-06-20 07:03:48 on dist-test-slave-wvz3"
server_key: "7444c208f70e40da16bf1162b9c74c13"
server_key_iv: "fdad3b26e4cb378283ececc88404cf25"
server_key_version: "encryptionkey@0"
I20260620 07:03:48.222594 16020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:03:48.226322 16034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:48.227926 16020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:48.228109 16020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "18f716a7d81048f89ff4fbf2e8ef5010"
format_stamp: "Formatted at 2026-06-20 07:03:48 on dist-test-slave-wvz3"
server_key: "7444c208f70e40da16bf1162b9c74c13"
server_key_iv: "fdad3b26e4cb378283ececc88404cf25"
server_key_version: "encryptionkey@0"
I20260620 07:03:48.228322 16020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:48.258251 16020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:48.262959 16020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:48.263247 16020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:48.264309 16020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:48.265676 16020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:48.265769 16020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:48.265877 16020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:48.265951 16020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:48.297047 16020 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42765
I20260620 07:03:48.297076 16146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42765 every 8 connection(s)
I20260620 07:03:48.297902 16020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:48.307328 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16020
I20260620 07:03:48.307546 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:48.308099 11591 external_mini_cluster.cc:1428] Setting key 5e6ee822dd246af03c953b4893ed6639
Jun 20 07:03:48 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939028, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:48.318264 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.300472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:38927 (local address 127.11.81.254:35185)
0620 07:03:48.300638 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.300642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.301767 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 07:03:48.303575 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.303582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.303662 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.303750 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:48.305274 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.306027 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.308044 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.308596 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.308735 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.312457 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:48.312463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:48.312466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:48.312503 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:48.315104 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.315998 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.316002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.316004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.316132 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.316555 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.316559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.316561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.316836 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:48.316982 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.317695 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.317931 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:03:48.319207 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.300865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35185 (local address 127.11.81.195:38927)
0620 07:03:48.301586 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.301630 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.303264 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.303991 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.303999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.304520 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:48.305094 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.305111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.306171 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.306174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.307878 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.307893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.308854 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.308861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.309147 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.309873 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:48.309912 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:48.312248 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:48.315252 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:48.315260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:48.315274 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:48.315805 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:48.316245 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:48.316249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:48.316251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:48.316442 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:48.316963 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:48.316970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:48.317302 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.317554 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.318106 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":289,"threads_started":1}
I20260620 07:03:48.320704 16147 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35185
I20260620 07:03:48.321044 16147 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:48.321695 16147 heartbeater.cc:511] Master 127.11.81.254:35185 requested a full tablet report, sending...
I20260620 07:03:48.323271 15694 ts_manager.cc:194] Registered new tserver with Master: 18f716a7d81048f89ff4fbf2e8ef5010 (127.11.81.195:42765)
I20260620 07:03:48.323962 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:48.324258 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:38927
I20260620 07:03:48.341440 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.329064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.11.81.254:35185)
0620 07:03:48.329224 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.329228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.329603 (+ 375us) server_negotiation.cc:407] Connection header received
0620 07:03:48.329736 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.329740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.329797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.329879 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:48.331242 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.332095 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.333472 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.333972 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.334076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.336380 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:48.336385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:48.336389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:48.336423 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:48.338878 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.339623 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.339634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.339636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.339771 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.340178 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.340181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.340184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.340507 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:48.340699 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.341025 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.341208 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:03:48.348354 15882 heartbeater.cc:503] Master 127.11.81.254:35185 was elected leader, sending a full tablet report...
I20260620 07:03:48.345341 15694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:48.352542 15694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:48.374464 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.365102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.11.81.193:43669)
0620 07:03:48.366322 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.366331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.366358 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:03:48.366456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.366463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.366736 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.367641 (+ 905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:48.367924 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.369283 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.371908 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.373294 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.373438 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.373683 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.373974 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.374142 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":878,"threads_started":1}
I20260620 07:03:48.374459 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.364865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:43669 (local address 127.0.0.1:57856)
0620 07:03:48.365621 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.365666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.365868 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.366999 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.367006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.367047 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:48.367420 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.367433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.369425 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.369430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.371718 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.371732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.373513 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.373520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.373732 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.373784 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.373908 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.374093 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260620 07:03:48.375593 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.366543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42765 (local address 127.0.0.1:54246)
0620 07:03:48.367032 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.367048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.367157 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.368570 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.368576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.368603 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:48.369061 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.369073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.370917 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.370922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.373082 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.373097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.374832 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.374838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.375007 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.375037 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.375100 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.375217 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":65,"thread_start_us":908,"threads_started":1}
I20260620 07:03:48.376212 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.367114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.11.81.195:42765)
0620 07:03:48.367939 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.367949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.367992 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:03:48.368097 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.368106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.368347 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.369259 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:48.369547 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.370775 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.373245 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.374598 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.375418 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.375649 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.375727 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.375881 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":472,"threads_started":1}
I20260620 07:03:48.377405 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.365964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35931 (local address 127.0.0.1:47964)
0620 07:03:48.366439 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.366454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.366557 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.370397 (+ 3840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.370402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.370426 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:48.370804 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.370815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.372511 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.372516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.374391 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.374404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.376590 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.376597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.376758 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.376790 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.376882 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.377009 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":1,"thread_start_us":275,"threads_started":1}
I20260620 07:03:48.377449 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.366443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.11.81.194:35931)
0620 07:03:48.369836 (+ 3393us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.369844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.369870 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:48.369969 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.369977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.370195 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.370385 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:48.371002 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.372394 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.374573 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.376329 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.376502 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.376710 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.376925 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.377133 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3237,"thread_start_us":3095,"threads_started":1}
I20260620 07:03:48.378922 15817 tablet_service.cc:1467] Processing CreateTablet for tablet 02e4af60cf024cce9ad65257f61383ba (DEFAULT_TABLE table=test-table [id=953443fcf64245359f599c2538045e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:48.380458 15817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e4af60cf024cce9ad65257f61383ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:48.380806 15950 tablet_service.cc:1467] Processing CreateTablet for tablet 02e4af60cf024cce9ad65257f61383ba (DEFAULT_TABLE table=test-table [id=953443fcf64245359f599c2538045e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:48.382128 15950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e4af60cf024cce9ad65257f61383ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:48.382335 16082 tablet_service.cc:1467] Processing CreateTablet for tablet 02e4af60cf024cce9ad65257f61383ba (DEFAULT_TABLE table=test-table [id=953443fcf64245359f599c2538045e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:48.383387 16082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e4af60cf024cce9ad65257f61383ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:48.394053 16165 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Bootstrap starting.
I20260620 07:03:48.397944 16165 tablet_bootstrap.cc:654] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:48.399235 16165 log.cc:824] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:48.400070 16167 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Bootstrap starting.
I20260620 07:03:48.401014 16166 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Bootstrap starting.
I20260620 07:03:48.403697 16167 tablet_bootstrap.cc:654] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:48.403822 16165 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: No bootstrap required, opened a new log
I20260620 07:03:48.404150 16165 ts_tablet_manager.cc:1397] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:03:48.404836 16166 tablet_bootstrap.cc:654] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:48.404989 16167 log.cc:824] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:48.405975 16166 log.cc:824] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:48.411913 16166 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: No bootstrap required, opened a new log
I20260620 07:03:48.412176 16166 ts_tablet_manager.cc:1397] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 07:03:48.413491 16167 tablet_bootstrap.cc:492] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: No bootstrap required, opened a new log
I20260620 07:03:48.413805 16167 ts_tablet_manager.cc:1397] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260620 07:03:48.414050 16165 raft_consensus.cc:348] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.414521 16165 raft_consensus.cc:374] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:48.414628 16165 raft_consensus.cc:729] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f716a7d81048f89ff4fbf2e8ef5010, State: Initialized, Role: FOLLOWER
I20260620 07:03:48.415264 16165 consensus_queue.cc:260] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.417922 16147 heartbeater.cc:503] Master 127.11.81.254:35185 was elected leader, sending a full tablet report...
I20260620 07:03:48.418598 16165 ts_tablet_manager.cc:1428] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 07:03:48.420281 16166 raft_consensus.cc:348] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.420725 16166 raft_consensus.cc:374] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:48.420822 16166 raft_consensus.cc:729] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5595b749597240e98756fb25a9a60701, State: Initialized, Role: FOLLOWER
I20260620 07:03:48.421492 16166 consensus_queue.cc:260] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.423542 16166 ts_tablet_manager.cc:1428] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260620 07:03:48.424503 16167 raft_consensus.cc:348] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.424980 16167 raft_consensus.cc:374] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:48.425086 16167 raft_consensus.cc:729] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3828edbf3446598a12a73eb047fd6b, State: Initialized, Role: FOLLOWER
I20260620 07:03:48.425693 16167 consensus_queue.cc:260] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.426800 16015 heartbeater.cc:503] Master 127.11.81.254:35185 was elected leader, sending a full tablet report...
I20260620 07:03:48.427428 16167 ts_tablet_manager.cc:1428] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260620 07:03:48.547366 16173 raft_consensus.cc:482] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:48.547680 16173 raft_consensus.cc:504] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
W20260620 07:03:48.549286 16148 tablet.cc:2369] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:48.549572 16173 leader_election.cc:288] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5595b749597240e98756fb25a9a60701 (127.11.81.193:43669), 18f716a7d81048f89ff4fbf2e8ef5010 (127.11.81.195:42765)
I20260620 07:03:48.556807 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.549949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:44967 (local address 127.11.81.193:43669)
0620 07:03:48.550237 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.550244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.550749 (+ 505us) server_negotiation.cc:407] Connection header received
0620 07:03:48.550935 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.550940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.551019 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.551113 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:48.551759 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.552693 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.554734 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.555803 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.555925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.555979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.556403 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.556543 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:03:48.556954 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.549958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:43669 (local address 127.11.81.194:44967)
0620 07:03:48.550634 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.550650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.550801 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.551169 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.551174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.551206 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:48.551604 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.551617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.552839 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.552845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.554588 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.554602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.556006 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.556014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.556217 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.556254 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.556359 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.556566 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":343,"threads_started":1}
I20260620 07:03:48.557674 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.550942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:37499 (local address 127.11.81.195:42765)
0620 07:03:48.551118 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.551123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.551445 (+ 322us) server_negotiation.cc:407] Connection header received
0620 07:03:48.551572 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.551577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.551649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.551731 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:03:48.552406 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.553400 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.555385 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.556526 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.556673 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.556743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.557331 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.557459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:03:48.558238 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.550719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42765 (local address 127.11.81.194:37499)
0620 07:03:48.551347 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:48.551368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:48.551470 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:48.551760 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:48.551764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:48.551788 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:03:48.552230 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.552242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.553558 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.553562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.555196 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:48.555210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.556881 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.556891 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:48.557138 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.557170 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:03:48.557844 (+ 674us) client_negotiation.cc:240] Negotiation successful
0620 07:03:48.557981 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":400,"threads_started":1}
I20260620 07:03:48.558010 15837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e4af60cf024cce9ad65257f61383ba" candidate_uuid: "ff3828edbf3446598a12a73eb047fd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5595b749597240e98756fb25a9a60701" is_pre_election: true
I20260620 07:03:48.558607 15837 raft_consensus.cc:2454] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3828edbf3446598a12a73eb047fd6b in term 0.
I20260620 07:03:48.559343 15906 leader_election.cc:302] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5595b749597240e98756fb25a9a60701, ff3828edbf3446598a12a73eb047fd6b; no voters:
I20260620 07:03:48.559260 16102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e4af60cf024cce9ad65257f61383ba" candidate_uuid: "ff3828edbf3446598a12a73eb047fd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" is_pre_election: true
I20260620 07:03:48.559759 16102 raft_consensus.cc:2454] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3828edbf3446598a12a73eb047fd6b in term 0.
I20260620 07:03:48.559767 16173 raft_consensus.cc:2781] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:48.559932 16173 raft_consensus.cc:482] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:48.560045 16173 raft_consensus.cc:3037] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 0 FOLLOWER]: Advancing to term 1
W20260620 07:03:48.562348 16016 tablet.cc:2369] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:48.562326 16173 raft_consensus.cc:504] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
I20260620 07:03:48.563066 16173 leader_election.cc:288] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [CANDIDATE]: Term 1 election: Requested vote from peers 5595b749597240e98756fb25a9a60701 (127.11.81.193:43669), 18f716a7d81048f89ff4fbf2e8ef5010 (127.11.81.195:42765)
I20260620 07:03:48.563529 15837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e4af60cf024cce9ad65257f61383ba" candidate_uuid: "ff3828edbf3446598a12a73eb047fd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5595b749597240e98756fb25a9a60701"
I20260620 07:03:48.563613 16102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e4af60cf024cce9ad65257f61383ba" candidate_uuid: "ff3828edbf3446598a12a73eb047fd6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f716a7d81048f89ff4fbf2e8ef5010"
I20260620 07:03:48.563761 15837 raft_consensus.cc:3037] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:48.563783 16102 raft_consensus.cc:3037] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:48.566825 15837 raft_consensus.cc:2454] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3828edbf3446598a12a73eb047fd6b in term 1.
I20260620 07:03:48.566825 16102 raft_consensus.cc:2454] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3828edbf3446598a12a73eb047fd6b in term 1.
I20260620 07:03:48.567234 15906 leader_election.cc:302] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f716a7d81048f89ff4fbf2e8ef5010, ff3828edbf3446598a12a73eb047fd6b; no voters:
I20260620 07:03:48.567600 16173 raft_consensus.cc:2781] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:48.568022 16173 raft_consensus.cc:686] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 1 LEADER]: Becoming Leader. State: Replica: ff3828edbf3446598a12a73eb047fd6b, State: Running, Role: LEADER
I20260620 07:03:48.568575 16173 consensus_queue.cc:237] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } }
W20260620 07:03:48.569502 15883 tablet.cc:2369] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:48.573571 15694 catalog_manager.cc:5495] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b reported cstate change: term changed from 0 to 1, leader changed from <none> to ff3828edbf3446598a12a73eb047fd6b (127.11.81.194). New cstate: current_term: 1 leader_uuid: "ff3828edbf3446598a12a73eb047fd6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff3828edbf3446598a12a73eb047fd6b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 } health_report { overall_health: UNKNOWN } } }
I20260620 07:03:48.615602 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.610180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.11.81.194:35931)
0620 07:03:48.610356 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.610361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.610467 (+ 106us) server_negotiation.cc:407] Connection header received
0620 07:03:48.610675 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.610679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.610788 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.610911 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:48.611256 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.611994 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.613291 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.613774 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.613878 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.614402 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:48.614994 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:48.615111 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.615270 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.615376 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:03:48.624048 15837 raft_consensus.cc:1261] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 1 FOLLOWER]: Refusing update from remote peer ff3828edbf3446598a12a73eb047fd6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:48.624094 16102 raft_consensus.cc:1261] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 1 FOLLOWER]: Refusing update from remote peer ff3828edbf3446598a12a73eb047fd6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:48.624985 16173 consensus_queue.cc:1035] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5595b749597240e98756fb25a9a60701" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 43669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:48.625309 16176 consensus_queue.cc:1035] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18f716a7d81048f89ff4fbf2e8ef5010" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:48.637065 16179 mvcc.cc:205] Tried to move back new op lower bound from 7298822261233250304 to 7298822261021044736. Current Snapshot: MvccSnapshot[applied={T|T < 7298822261233250304}]
I20260620 07:03:48.649995 15694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:48.650291 15694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:48.653874 15694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93a4345e7e524869b765c593977b7e8d: Sending DeleteTablet for 3 replicas of tablet 02e4af60cf024cce9ad65257f61383ba
I20260620 07:03:48.655114 15817 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e4af60cf024cce9ad65257f61383ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:48 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:57856
I20260620 07:03:48.655272 16082 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e4af60cf024cce9ad65257f61383ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:48 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:54246
I20260620 07:03:48.655629 15950 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e4af60cf024cce9ad65257f61383ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:48 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:47964
I20260620 07:03:48.656091 16188 tablet_replica.cc:330] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: stopping tablet replica
I20260620 07:03:48.656442 16190 tablet_replica.cc:330] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: stopping tablet replica
I20260620 07:03:48.656565 16188 raft_consensus.cc:2229] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:48.656857 16190 raft_consensus.cc:2229] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:48.657117 16188 raft_consensus.cc:2258] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:48.657594 16190 raft_consensus.cc:2258] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:48.659233 16188 ts_tablet_manager.cc:1905] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:48.660123 16190 ts_tablet_manager.cc:1905] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:48.660490 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15755
I20260620 07:03:48.666249 16188 ts_tablet_manager.cc:1918] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:48.666520 16188 log.cc:1196] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/02e4af60cf024cce9ad65257f61383ba
I20260620 07:03:48.667419 16188 ts_tablet_manager.cc:1939] T 02e4af60cf024cce9ad65257f61383ba P 5595b749597240e98756fb25a9a60701: Deleting consensus metadata
I20260620 07:03:48.667560 16190 ts_tablet_manager.cc:1918] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:48.667780 16190 log.cc:1196] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/02e4af60cf024cce9ad65257f61383ba
I20260620 07:03:48.668331 16190 ts_tablet_manager.cc:1939] T 02e4af60cf024cce9ad65257f61383ba P ff3828edbf3446598a12a73eb047fd6b: Deleting consensus metadata
I20260620 07:03:48.669423 15681 catalog_manager.cc:4841] TS 5595b749597240e98756fb25a9a60701 (127.11.81.193:43669): tablet 02e4af60cf024cce9ad65257f61383ba (table test-table [id=953443fcf64245359f599c2538045e9b]) successfully deleted
I20260620 07:03:48.669977 15681 catalog_manager.cc:4841] TS ff3828edbf3446598a12a73eb047fd6b (127.11.81.194:35931): tablet 02e4af60cf024cce9ad65257f61383ba (table test-table [id=953443fcf64245359f599c2538045e9b]) successfully deleted
I20260620 07:03:48.672444 16189 tablet_replica.cc:330] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: stopping tablet replica
I20260620 07:03:48.672962 16189 raft_consensus.cc:2229] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:48.673511 16189 raft_consensus.cc:2258] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:48.675211 16189 ts_tablet_manager.cc:1905] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:03:48 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:03:48.681669 16189 ts_tablet_manager.cc:1918] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:48.681821 16189 log.cc:1196] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/02e4af60cf024cce9ad65257f61383ba
I20260620 07:03:48.682240 16189 ts_tablet_manager.cc:1939] T 02e4af60cf024cce9ad65257f61383ba P 18f716a7d81048f89ff4fbf2e8ef5010: Deleting consensus metadata
I20260620 07:03:48.684223 15681 catalog_manager.cc:4841] TS 18f716a7d81048f89ff4fbf2e8ef5010 (127.11.81.195:42765): tablet 02e4af60cf024cce9ad65257f61383ba (table test-table [id=953443fcf64245359f599c2538045e9b]) successfully deleted
I20260620 07:03:48.688827 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.663368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.11.81.193:43669)
0620 07:03:48.663560 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.663566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.671835 (+ 8269us) server_negotiation.cc:407] Connection header received
0620 07:03:48.672045 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.672051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.672135 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.672208 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:48.673931 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.675062 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.676792 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.677517 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.677641 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.682637 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:48.682657 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:48.682676 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:48.682723 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:48.685735 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.686698 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.686706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.686708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.686860 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.687264 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.687267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.687269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.687696 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:48.687975 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.688329 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.688510 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:03:48.689572 15877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:48.839883 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15755
I20260620 07:03:48.860683 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15888
Jun 20 07:03:48 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:03:48.881356 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:48.861760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.11.81.194:35931)
0620 07:03:48.861953 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:48.861959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:48.861980 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:03:48.862143 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:48.862150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:48.862229 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:48.862321 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:48.864243 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.865480 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.867436 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:48.868093 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:48.868212 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:48.873650 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:48.873673 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:48.873694 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:48.873752 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:48.878075 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.878899 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.878909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.878913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.879112 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:48.879652 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:48.879657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:48.879660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:48.880119 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:48.880456 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:48.880703 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 07:03:48.880942 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:03:48.882135 16010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:48.996862 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15888
I20260620 07:03:49.012244 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16020
Jun 20 07:03:49 dist-test-slave-wvz3 krb5kdc[15650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:03:49.027561 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:49.013121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.11.81.195:42765)
0620 07:03:49.013284 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:49.013289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:49.013307 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:49.013361 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:49.013365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:49.013426 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:49.013520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:49.014843 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:49.015656 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:49.017047 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:49.017450 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:49.017548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:49.022186 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:49.022201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:49.022217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:49.022256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:49.024837 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:49.025514 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:49.025521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:49.025523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:49.025650 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:49.025929 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:49.025932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:49.025934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:49.026310 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:49.026543 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:49.026947 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 07:03:49.027161 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:03:49.028227 16142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:49.116772 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16020
I20260620 07:03:49.129464 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 15663
I20260620 07:03:49.130637 15724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:49.237824 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 15663
2026-06-20T07:03:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4145 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T07:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:49Z Disabled control of system clock
I20260620 07:03:49.299270 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41849
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:41849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:03:49.618434 16199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:49.618733 16199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:49.618824 16199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:49.626109 16199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:49.626219 16199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:49.626255 16199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:49.626309 16199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:49.626353 16199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:49.635633 16199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:41849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:49.637310 16199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:49.639535 16199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:49.651698 16207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:49.651803 16204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:49.651722 16205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:49.652585 16199 server_base.cc:1034] running on GCE node
I20260620 07:03:49.653401 16199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:49.654960 16199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:49.656216 16199 hybrid_clock.cc:648] HybridClock initialized: now 1781939029656151 us; error 61 us; skew 500 ppm
I20260620 07:03:49.656620 16199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:49.665180 16199 webserver.cc:460] Webserver started at http://127.11.81.254:45521/ using document root <none> and password file <none>
I20260620 07:03:49.665995 16199 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:49.666109 16199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:49.666435 16199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:49.669520 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "43e691e0c7164e18a71d61cc33862034"
format_stamp: "Formatted at 2026-06-20 07:03:49 on dist-test-slave-wvz3"
server_key: "3fa48aa72a2a5db3bd3f3a42dcab3399"
server_key_iv: "af509ea692184a038397c4599dbef255"
server_key_version: "encryptionkey@0"
I20260620 07:03:49.670337 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "43e691e0c7164e18a71d61cc33862034"
format_stamp: "Formatted at 2026-06-20 07:03:49 on dist-test-slave-wvz3"
server_key: "3fa48aa72a2a5db3bd3f3a42dcab3399"
server_key_iv: "af509ea692184a038397c4599dbef255"
server_key_version: "encryptionkey@0"
I20260620 07:03:49.675935 16199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:03:49.679639 16213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:49.681300 16199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:49.681522 16199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "43e691e0c7164e18a71d61cc33862034"
format_stamp: "Formatted at 2026-06-20 07:03:49 on dist-test-slave-wvz3"
server_key: "3fa48aa72a2a5db3bd3f3a42dcab3399"
server_key_iv: "af509ea692184a038397c4599dbef255"
server_key_version: "encryptionkey@0"
I20260620 07:03:49.681761 16199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:49.688061 16199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:49.688915 16199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:49.689206 16199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:49.707110 16199 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:41849
I20260620 07:03:49.707129 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:41849 every 8 connection(s)
I20260620 07:03:49.708493 16199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:49.712292 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:49.717949 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16199
I20260620 07:03:49.718147 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:49.718788 11591 external_mini_cluster.cc:1428] Setting key 158ea08d0000779997151068f68119b3
I20260620 07:03:49.723443 16265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: Bootstrap starting.
I20260620 07:03:49.726637 16265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:49.727193 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:49.720911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.11.81.254:41849)
0620 07:03:49.721943 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:49.721955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:49.721998 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:03:49.723682 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:49.723709 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:49.724139 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:49.724498 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:49.725552 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:49.725559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:49.725573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:49.725587 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:49.725652 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:49.725668 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:49.725875 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:49.726008 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 07:03:49.726253 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260620 07:03:49.727851 16265 log.cc:824] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:49.730842 16265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: No bootstrap required, opened a new log
I20260620 07:03:49.738579 16265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } }
I20260620 07:03:49.738976 16265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:49.739054 16265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e691e0c7164e18a71d61cc33862034, State: Initialized, Role: FOLLOWER
I20260620 07:03:49.739627 16265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } }
I20260620 07:03:49.739833 16265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:49.739939 16265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:49.740067 16265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:49.742230 16265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } }
I20260620 07:03:49.742707 16265 leader_election.cc:302] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43e691e0c7164e18a71d61cc33862034; no voters:
I20260620 07:03:49.743382 16265 leader_election.cc:288] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:49.743526 16270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:49.744639 16270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [term 1 LEADER]: Becoming Leader. State: Replica: 43e691e0c7164e18a71d61cc33862034, State: Running, Role: LEADER
I20260620 07:03:49.745199 16270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } }
I20260620 07:03:49.745642 16265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:49.747957 16271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43e691e0c7164e18a71d61cc33862034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } } }
I20260620 07:03:49.747995 16272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43e691e0c7164e18a71d61cc33862034. Latest consensus state: current_term: 1 leader_uuid: "43e691e0c7164e18a71d61cc33862034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e691e0c7164e18a71d61cc33862034" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41849 } } }
I20260620 07:03:49.748330 16271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:49.748368 16272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:49.750378 16276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:49.755826 16276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:49.765209 16276 catalog_manager.cc:1347] Generated new cluster ID: 77e7bd2a205f41a0a3083025fa8f394d
I20260620 07:03:49.765337 16276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:49.805622 16276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:49.806129 16276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:49.819110 16276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: Generated new TSK 0
I20260620 07:03:49.820026 16276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:03:49.833076 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--builtin_ntp_servers=127.11.81.212:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:03:50.158906 16289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:50.159219 16289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:50.159354 16289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:50.166833 16289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:50.166960 16289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:50.167107 16289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:50.176553 16289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:50.177888 16289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:50.179756 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:50.193022 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:50.192966 16295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:50.193280 16294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:50.194024 16289 server_base.cc:1034] running on GCE node
I20260620 07:03:50.194689 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:50.195751 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:50.196995 16289 hybrid_clock.cc:648] HybridClock initialized: now 1781939030196919 us; error 51 us; skew 500 ppm
I20260620 07:03:50.197324 16289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:50.199792 16289 webserver.cc:460] Webserver started at http://127.11.81.193:35319/ using document root <none> and password file <none>
I20260620 07:03:50.200587 16289 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:50.200686 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:50.201030 16289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:50.204134 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "f20e25b4bfe142fd98d1e751fd32951a"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "8a16856784d1c3bad6bb57bf2f178623"
server_key_iv: "78874c62eb9d366f87c277d9fa206567"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.205006 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "f20e25b4bfe142fd98d1e751fd32951a"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "8a16856784d1c3bad6bb57bf2f178623"
server_key_iv: "78874c62eb9d366f87c277d9fa206567"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.210521 16289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:03:50.214588 16303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.216136 16289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:50.216311 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "f20e25b4bfe142fd98d1e751fd32951a"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "8a16856784d1c3bad6bb57bf2f178623"
server_key_iv: "78874c62eb9d366f87c277d9fa206567"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.216550 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:50.222450 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:50.223311 16289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:50.223579 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:50.224560 16289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:50.225980 16289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:50.226074 16289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.226189 16289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:50.226256 16289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.256868 16289 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:34389
I20260620 07:03:50.256920 16415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:34389 every 8 connection(s)
I20260620 07:03:50.258213 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:50.262368 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16289
I20260620 07:03:50.262571 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:50.263458 11591 external_mini_cluster.cc:1428] Setting key a03caf4daefbe990fc917d95053dac09
I20260620 07:03:50.266433 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--builtin_ntp_servers=127.11.81.212:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:03:50.268075 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:50.261056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:35457 (local address 127.11.81.254:41849)
0620 07:03:50.261990 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:50.261997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:50.263396 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 07:03:50.265542 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:50.265548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:50.265610 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:50.265706 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:50.267226 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:50.267230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:50.267234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:50.267250 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:50.267353 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:50.267367 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:50.267474 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:50.267666 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:03:50.267733 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":683,"threads_started":1}
I20260620 07:03:50.269059 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:50.261799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41849 (local address 127.11.81.193:35457)
0620 07:03:50.263139 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:50.263203 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:50.265230 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:50.265997 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:50.266007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:50.266114 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:50.266911 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:50.266914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:50.266956 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:50.267052 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:50.267457 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:50.267468 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:50.267476 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:03:50.267746 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 07:03:50.268048 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":335,"threads_started":1}
I20260620 07:03:50.270712 16416 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41849
I20260620 07:03:50.271023 16416 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:50.271652 16416 heartbeater.cc:511] Master 127.11.81.254:41849 requested a full tablet report, sending...
I20260620 07:03:50.273772 16230 ts_manager.cc:194] Registered new tserver with Master: f20e25b4bfe142fd98d1e751fd32951a (127.11.81.193:34389)
W20260620 07:03:50.583163 16420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:50.583509 16420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:50.583657 16420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:50.591145 16420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:50.591253 16420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:50.591418 16420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:50.600540 16420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:50.601809 16420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:50.603610 16420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:50.616099 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:50.616115 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:50.616204 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:50.617269 16420 server_base.cc:1034] running on GCE node
I20260620 07:03:50.617865 16420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:50.618844 16420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:50.620041 16420 hybrid_clock.cc:648] HybridClock initialized: now 1781939030619985 us; error 50 us; skew 500 ppm
I20260620 07:03:50.620378 16420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:50.622582 16420 webserver.cc:460] Webserver started at http://127.11.81.194:38411/ using document root <none> and password file <none>
I20260620 07:03:50.623270 16420 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:50.623412 16420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:50.623734 16420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:50.626514 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "3ab833e4a1114741a0f1fd146c8efb6d"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "cc6e4149e63ac9e01e8adab33c0472a6"
server_key_iv: "5aa52ab612ed26ffc27deae6cbfec510"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.627319 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "3ab833e4a1114741a0f1fd146c8efb6d"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "cc6e4149e63ac9e01e8adab33c0472a6"
server_key_iv: "5aa52ab612ed26ffc27deae6cbfec510"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.632445 16420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:03:50.636215 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.637665 16420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:50.637837 16420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "3ab833e4a1114741a0f1fd146c8efb6d"
format_stamp: "Formatted at 2026-06-20 07:03:50 on dist-test-slave-wvz3"
server_key: "cc6e4149e63ac9e01e8adab33c0472a6"
server_key_iv: "5aa52ab612ed26ffc27deae6cbfec510"
server_key_version: "encryptionkey@0"
I20260620 07:03:50.638051 16420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:50.644840 16420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:50.645663 16420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:50.645920 16420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:50.646865 16420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:50.648412 16420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:50.648506 16420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.648612 16420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:50.648676 16420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:50.679610 16420 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36917
I20260620 07:03:50.679674 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36917 every 8 connection(s)
I20260620 07:03:50.680914 16420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:50.686285 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16420
I20260620 07:03:50.686456 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:50.687007 11591 external_mini_cluster.cc:1428] Setting key e6446b63cc10e3ca34a0f099162e588c
I20260620 07:03:50.689590 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:50.683445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:59887 (local address 127.11.81.254:41849)
0620 07:03:50.683660 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:50.683666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:50.685215 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 07:03:50.687362 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:50.687368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:50.687442 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:50.687532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:50.688909 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:50.688915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:50.688919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:50.688935 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:50.688966 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:50.688978 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:50.689086 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:50.689331 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 07:03:50.689385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:03:50.690217 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--builtin_ntp_servers=127.11.81.212:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:03:50.690901 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:50.684042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41849 (local address 127.11.81.194:59887)
0620 07:03:50.685047 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:50.685091 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:50.686993 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:50.687789 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:50.687798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:50.687913 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:50.688549 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:50.688553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:50.688605 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:50.688721 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:50.689068 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:50.689078 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:50.689085 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:03:50.689370 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 07:03:50.689648 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":285,"threads_started":1}
I20260620 07:03:50.692407 16547 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41849
I20260620 07:03:50.692724 16547 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:50.693446 16547 heartbeater.cc:511] Master 127.11.81.254:41849 requested a full tablet report, sending...
I20260620 07:03:50.695055 16230 ts_manager.cc:194] Registered new tserver with Master: 3ab833e4a1114741a0f1fd146c8efb6d (127.11.81.194:36917)
W20260620 07:03:51.009361 16550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:51.009665 16550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:51.009753 16550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:51.017197 16550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:51.017319 16550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:51.017431 16550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:03:51.027171 16550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:41849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:51.028524 16550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:51.030246 16550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:51.043561 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:51.043524 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:51.043582 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:51.044345 16550 server_base.cc:1034] running on GCE node
I20260620 07:03:51.044925 16550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:51.045933 16550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:51.047139 16550 hybrid_clock.cc:648] HybridClock initialized: now 1781939031047085 us; error 50 us; skew 500 ppm
I20260620 07:03:51.047500 16550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:51.049903 16550 webserver.cc:460] Webserver started at http://127.11.81.195:33449/ using document root <none> and password file <none>
I20260620 07:03:51.050597 16550 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:51.050684 16550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:51.051007 16550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:51.054069 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "5a9713c16d014ed49e7e3ef94740ee5a"
format_stamp: "Formatted at 2026-06-20 07:03:51 on dist-test-slave-wvz3"
server_key: "268f4c86cea2f998f94172bcf1bca491"
server_key_iv: "43b3e4ea2de6d9cd2ad1ed56df129c1e"
server_key_version: "encryptionkey@0"
I20260620 07:03:51.054850 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "5a9713c16d014ed49e7e3ef94740ee5a"
format_stamp: "Formatted at 2026-06-20 07:03:51 on dist-test-slave-wvz3"
server_key: "268f4c86cea2f998f94172bcf1bca491"
server_key_iv: "43b3e4ea2de6d9cd2ad1ed56df129c1e"
server_key_version: "encryptionkey@0"
I20260620 07:03:51.060348 16550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:03:51.064086 16564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:51.065580 16550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:51.065747 16550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "5a9713c16d014ed49e7e3ef94740ee5a"
format_stamp: "Formatted at 2026-06-20 07:03:51 on dist-test-slave-wvz3"
server_key: "268f4c86cea2f998f94172bcf1bca491"
server_key_iv: "43b3e4ea2de6d9cd2ad1ed56df129c1e"
server_key_version: "encryptionkey@0"
I20260620 07:03:51.065960 16550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:51.071667 16550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:51.072474 16550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:51.072731 16550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:51.073652 16550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:51.075062 16550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:51.075148 16550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:51.075258 16550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:51.075354 16550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 07:03:51.106087 16550 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:43289
I20260620 07:03:51.106137 16676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:43289 every 8 connection(s)
I20260620 07:03:51.107434 16550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:03:51.109786 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16550
I20260620 07:03:51.109958 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:03:51.110456 11591 external_mini_cluster.cc:1428] Setting key 0ca566ace488d3b2d36b5896db968ebb
I20260620 07:03:51.116397 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.110090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:36671 (local address 127.11.81.254:41849)
0620 07:03:51.110311 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.110318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.112084 (+ 1766us) server_negotiation.cc:407] Connection header received
0620 07:03:51.114128 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.114133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.114192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.114282 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.115685 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.115689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.115693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.115762 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.115796 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.115809 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.115918 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.116056 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.116106 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:03:51.117405 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.110663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:41849 (local address 127.11.81.195:36671)
0620 07:03:51.111894 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.111953 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.113849 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.114527 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.114537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.114642 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.115364 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.115366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.115408 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.115506 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.115889 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.115895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.115900 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.116114 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.116394 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":332,"threads_started":1}
I20260620 07:03:51.118847 16677 heartbeater.cc:347] Connected to a master server at 127.11.81.254:41849
I20260620 07:03:51.119138 16677 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:51.119788 16677 heartbeater.cc:511] Master 127.11.81.254:41849 requested a full tablet report, sending...
I20260620 07:03:51.121254 16230 ts_manager.cc:194] Registered new tserver with Master: 5a9713c16d014ed49e7e3ef94740ee5a (127.11.81.195:43289)
I20260620 07:03:51.126291 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:03:51.133122 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.130603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.11.81.254:41849)
0620 07:03:51.130784 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.130790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.131256 (+ 466us) server_negotiation.cc:407] Connection header received
0620 07:03:51.131443 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.131447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.131504 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.131595 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.132511 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.132515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.132518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.132533 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.132565 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.132577 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.132681 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.132848 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.132896 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:03:51.137362 16230 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:03:51.144385 16230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:03:51.178815 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.157298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:58996)
0620 07:03:51.168030 (+ 10732us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.168090 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.168323 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.176427 (+ 8104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.176439 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.176574 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.177423 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.177426 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.177459 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.177543 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.178063 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.178081 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.178090 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.178283 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.178374 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10377,"thread_start_us":19591,"threads_started":1}
I20260620 07:03:51.179394 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.157477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.11.81.193:34389)
0620 07:03:51.175094 (+ 17617us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.175102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.175136 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:03:51.175218 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.175227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.176209 (+ 982us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.176395 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.177733 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.177743 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.177757 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.177789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.177851 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.177872 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.178006 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.178964 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.179080 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17465,"thread_start_us":17141,"threads_started":1}
I20260620 07:03:51.179898 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.157443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:43289 (local address 127.0.0.1:40408)
0620 07:03:51.175968 (+ 18525us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.175991 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.176123 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.177565 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.177570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.177596 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.178445 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.178449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.178465 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.178513 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.179149 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.179155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.179160 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.179315 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.179363 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18347,"thread_start_us":10053,"threads_started":1}
I20260620 07:03:51.179893 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.160507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.11.81.194:36917)
0620 07:03:51.175338 (+ 14831us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.175349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.176693 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 07:03:51.176960 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.176970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.177487 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.177632 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.178978 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.178986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.178997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.179013 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.179063 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.179085 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.179249 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.179480 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.179582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14614,"thread_start_us":14465,"threads_started":1}
I20260620 07:03:51.180812 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.159826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36917 (local address 127.0.0.1:59182)
0620 07:03:51.176576 (+ 16750us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.176592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.176786 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.177992 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.177997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.178020 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.178720 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.178733 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.178748 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.178787 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.179326 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.179330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.179335 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.179494 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.179544 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16639,"thread_start_us":15672,"threads_started":1}
I20260620 07:03:51.180819 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.157676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.11.81.195:43289)
0620 07:03:51.174768 (+ 17092us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.174780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.177056 (+ 2276us) server_negotiation.cc:407] Connection header received
0620 07:03:51.177129 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.177135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.177381 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.177537 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.178660 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.178669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.178684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.178702 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.178758 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.178778 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.180294 (+ 1516us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.180371 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.180464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16872,"thread_start_us":16690,"threads_started":1}
I20260620 07:03:51.183660 16351 tablet_service.cc:1467] Processing CreateTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de (DEFAULT_TABLE table=test-table [id=2af2404abd124d4883856e022616f5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:51.183884 16482 tablet_service.cc:1467] Processing CreateTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de (DEFAULT_TABLE table=test-table [id=2af2404abd124d4883856e022616f5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:51.184350 16612 tablet_service.cc:1467] Processing CreateTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de (DEFAULT_TABLE table=test-table [id=2af2404abd124d4883856e022616f5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:03:51.185127 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdd05bfb9ae4579aa719b8d2cbf83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:51.185144 16351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdd05bfb9ae4579aa719b8d2cbf83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:51.185571 16612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfdd05bfb9ae4579aa719b8d2cbf83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:51.197665 16695 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Bootstrap starting.
I20260620 07:03:51.199838 16697 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Bootstrap starting.
I20260620 07:03:51.200647 16696 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Bootstrap starting.
I20260620 07:03:51.202064 16695 tablet_bootstrap.cc:654] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:51.203958 16697 tablet_bootstrap.cc:654] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:51.204011 16695 log.cc:824] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:51.204650 16696 tablet_bootstrap.cc:654] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:51.205096 16697 log.cc:824] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:51.205863 16696 log.cc:824] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:51.207577 16695 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: No bootstrap required, opened a new log
I20260620 07:03:51.207718 16697 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: No bootstrap required, opened a new log
I20260620 07:03:51.207912 16695 ts_tablet_manager.cc:1397] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 07:03:51.207968 16697 ts_tablet_manager.cc:1397] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:03:51.215369 16696 tablet_bootstrap.cc:492] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: No bootstrap required, opened a new log
I20260620 07:03:51.215713 16696 ts_tablet_manager.cc:1397] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260620 07:03:51.215906 16697 raft_consensus.cc:348] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.216293 16697 raft_consensus.cc:374] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:51.216394 16697 raft_consensus.cc:729] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9713c16d014ed49e7e3ef94740ee5a, State: Initialized, Role: FOLLOWER
I20260620 07:03:51.217052 16697 consensus_queue.cc:260] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.218202 16677 heartbeater.cc:503] Master 127.11.81.254:41849 was elected leader, sending a full tablet report...
I20260620 07:03:51.218874 16697 ts_tablet_manager.cc:1428] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260620 07:03:51.218562 16695 raft_consensus.cc:348] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.219002 16695 raft_consensus.cc:374] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:51.219082 16695 raft_consensus.cc:729] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab833e4a1114741a0f1fd146c8efb6d, State: Initialized, Role: FOLLOWER
I20260620 07:03:51.219851 16695 consensus_queue.cc:260] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.221041 16547 heartbeater.cc:503] Master 127.11.81.254:41849 was elected leader, sending a full tablet report...
I20260620 07:03:51.221709 16695 ts_tablet_manager.cc:1428] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 07:03:51.226773 16696 raft_consensus.cc:348] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.227154 16696 raft_consensus.cc:374] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:51.227258 16696 raft_consensus.cc:729] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20e25b4bfe142fd98d1e751fd32951a, State: Initialized, Role: FOLLOWER
I20260620 07:03:51.227896 16696 consensus_queue.cc:260] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.228955 16416 heartbeater.cc:503] Master 127.11.81.254:41849 was elected leader, sending a full tablet report...
I20260620 07:03:51.229499 16696 ts_tablet_manager.cc:1428] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 07:03:51.260627 16417 tablet.cc:2369] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:51.359053 16678 tablet.cc:2369] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:03:51.432883 16548 tablet.cc:2369] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:51.601219 16702 raft_consensus.cc:482] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:51.601549 16702 raft_consensus.cc:504] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.603057 16703 raft_consensus.cc:482] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:51.603374 16702 leader_election.cc:288] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20e25b4bfe142fd98d1e751fd32951a (127.11.81.193:34389), 5a9713c16d014ed49e7e3ef94740ee5a (127.11.81.195:43289)
I20260620 07:03:51.603381 16703 raft_consensus.cc:504] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.606212 16703 leader_election.cc:288] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9713c16d014ed49e7e3ef94740ee5a (127.11.81.195:43289), 3ab833e4a1114741a0f1fd146c8efb6d (127.11.81.194:36917)
I20260620 07:03:51.607448 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.603891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46015 (local address 127.11.81.193:34389)
0620 07:03:51.604070 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.604077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.604927 (+ 850us) server_negotiation.cc:407] Connection header received
0620 07:03:51.605198 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.605204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.605269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.605387 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.606642 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.606647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.606651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.606669 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.606706 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.606723 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.606804 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.607023 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.607085 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:03:51.608031 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.604362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:43289 (local address 127.11.81.194:39385)
0620 07:03:51.605062 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.605076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.605224 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.605578 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.605583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.605611 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.606509 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.606513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.606531 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.606581 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.607351 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.607355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.607360 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.607486 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.607538 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":406,"threads_started":1}
I20260620 07:03:51.608157 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.603785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.11.81.194:46015)
0620 07:03:51.604792 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.604813 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.605030 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.605430 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.605434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.605458 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.606408 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.606411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.606430 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.606483 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.606890 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.606894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.606897 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.607740 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.607790 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":648,"threads_started":1}
I20260620 07:03:51.608753 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.603926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:39385 (local address 127.11.81.195:43289)
0620 07:03:51.604148 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.604154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.605147 (+ 993us) server_negotiation.cc:407] Connection header received
0620 07:03:51.605355 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.605360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.605421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.605506 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.607090 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.607095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.607099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.607116 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.607155 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.607170 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.607279 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.608434 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.608496 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:03:51.609264 16701 raft_consensus.cc:482] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:03:51.609062 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20e25b4bfe142fd98d1e751fd32951a" is_pre_election: true
I20260620 07:03:51.609643 16371 raft_consensus.cc:2454] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab833e4a1114741a0f1fd146c8efb6d in term 0.
I20260620 07:03:51.609563 16701 raft_consensus.cc:504] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.609673 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" is_pre_election: true
I20260620 07:03:51.610351 16438 leader_election.cc:302] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab833e4a1114741a0f1fd146c8efb6d, f20e25b4bfe142fd98d1e751fd32951a; no voters:
I20260620 07:03:51.610679 16632 raft_consensus.cc:2454] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ab833e4a1114741a0f1fd146c8efb6d in term 0.
I20260620 07:03:51.610785 16702 raft_consensus.cc:2781] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:03:51.610906 16702 raft_consensus.cc:482] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:03:51.610992 16702 raft_consensus.cc:3037] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:51.611565 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.605791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:60773 (local address 127.11.81.195:43289)
0620 07:03:51.607146 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.607153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.608327 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 07:03:51.608382 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.608387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.608441 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.608536 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.610618 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.610624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.610628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.610647 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.610698 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.610713 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.610828 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.611035 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.611086 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":1600,"threads_started":1}
I20260620 07:03:51.611627 16701 leader_election.cc:288] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f20e25b4bfe142fd98d1e751fd32951a (127.11.81.193:34389), 3ab833e4a1114741a0f1fd146c8efb6d (127.11.81.194:36917)
I20260620 07:03:51.612596 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.605800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:43289 (local address 127.11.81.193:60773)
0620 07:03:51.608097 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.608113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.608208 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.609420 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.609425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.609447 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.610165 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.610168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.610190 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.610258 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.610914 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.610918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.610922 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.611759 (+ 837us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.611815 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2190,"thread_start_us":1546,"threads_started":1}
I20260620 07:03:51.612799 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.606849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46003 (local address 127.11.81.194:36917)
0620 07:03:51.607005 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.607010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.608591 (+ 1581us) server_negotiation.cc:407] Connection header received
0620 07:03:51.608814 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.608819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.608871 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.608935 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.612093 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.612098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.612102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.612129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.612162 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.612181 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.612272 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.612491 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.612550 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:03:51.613538 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.606423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36917 (local address 127.11.81.193:46003)
0620 07:03:51.608509 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.608527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.608674 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.609115 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.609119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.609141 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.611878 (+ 2737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.611881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.611896 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.611936 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.612354 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.612358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.612362 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.613047 (+ 685us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.613097 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1958,"thread_start_us":1375,"threads_started":1}
I20260620 07:03:51.614226 16702 raft_consensus.cc:504] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.614548 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "f20e25b4bfe142fd98d1e751fd32951a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" is_pre_election: true
I20260620 07:03:51.615146 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.612096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.11.81.195:41789)
0620 07:03:51.612686 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.612702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.612781 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.613647 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.613652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.613675 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.614346 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.614349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.614365 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.614409 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.614771 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.614776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.614780 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.614869 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.614917 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489}
I20260620 07:03:51.615149 16502 raft_consensus.cc:2379] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f20e25b4bfe142fd98d1e751fd32951a in current term 1: Already voted for candidate 3ab833e4a1114741a0f1fd146c8efb6d in this term.
I20260620 07:03:51.615377 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20e25b4bfe142fd98d1e751fd32951a"
I20260620 07:03:51.615574 16371 raft_consensus.cc:3037] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:51.615656 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.612397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:41789 (local address 127.11.81.193:34389)
0620 07:03:51.612542 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.612547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.613243 (+ 696us) server_negotiation.cc:407] Connection header received
0620 07:03:51.613295 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.613300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.613395 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.613484 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.614553 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.614558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.614562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.614578 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.614608 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.614621 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.614721 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.615342 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.615397 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:03:51.615976 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "f20e25b4bfe142fd98d1e751fd32951a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" is_pre_election: true
I20260620 07:03:51.616247 16632 raft_consensus.cc:2454] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f20e25b4bfe142fd98d1e751fd32951a in term 0.
I20260620 07:03:51.616665 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.613187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:34779 (local address 127.11.81.194:36917)
0620 07:03:51.613517 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.613523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.614437 (+ 914us) server_negotiation.cc:407] Connection header received
0620 07:03:51.614634 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.614643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.614728 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.614798 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.615911 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.615916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.615920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.615938 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.615969 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.615983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.616070 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.616341 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.616396 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260620 07:03:51.616922 16304 leader_election.cc:302] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a9713c16d014ed49e7e3ef94740ee5a, f20e25b4bfe142fd98d1e751fd32951a; no voters: 3ab833e4a1114741a0f1fd146c8efb6d
I20260620 07:03:51.617338 16632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9713c16d014ed49e7e3ef94740ee5a"
I20260620 07:03:51.617476 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.613082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36917 (local address 127.11.81.195:34779)
0620 07:03:51.614348 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.614369 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:51.614507 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:51.614996 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:51.615000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:51.615021 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:51.615687 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:03:51.615690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:51.615706 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:03:51.615759 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:51.616157 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:51.616161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:51.616164 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:03:51.616974 (+ 810us) client_negotiation.cc:240] Negotiation successful
0620 07:03:51.617025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":834,"threads_started":1}
I20260620 07:03:51.617535 16632 raft_consensus.cc:3037] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:51.617658 16370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f20e25b4bfe142fd98d1e751fd32951a" is_pre_election: true
I20260620 07:03:51.617966 16702 leader_election.cc:288] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [CANDIDATE]: Term 1 election: Requested vote from peers f20e25b4bfe142fd98d1e751fd32951a (127.11.81.193:34389), 5a9713c16d014ed49e7e3ef94740ee5a (127.11.81.195:43289)
I20260620 07:03:51.618211 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfdd05bfb9ae4579aa719b8d2cbf83de" candidate_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" is_pre_election: true
I20260620 07:03:51.618428 16502 raft_consensus.cc:2379] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a9713c16d014ed49e7e3ef94740ee5a in current term 1: Already voted for candidate 3ab833e4a1114741a0f1fd146c8efb6d in this term.
I20260620 07:03:51.618947 16566 leader_election.cc:302] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9713c16d014ed49e7e3ef94740ee5a; no voters: 3ab833e4a1114741a0f1fd146c8efb6d, f20e25b4bfe142fd98d1e751fd32951a
I20260620 07:03:51.619716 16371 raft_consensus.cc:2454] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab833e4a1114741a0f1fd146c8efb6d in term 1.
I20260620 07:03:51.619930 16703 raft_consensus.cc:2741] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 07:03:51.620090 16438 leader_election.cc:302] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab833e4a1114741a0f1fd146c8efb6d, f20e25b4bfe142fd98d1e751fd32951a; no voters:
I20260620 07:03:51.620410 16702 raft_consensus.cc:2781] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:51.620630 16632 raft_consensus.cc:2454] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ab833e4a1114741a0f1fd146c8efb6d in term 1.
I20260620 07:03:51.620798 16701 raft_consensus.cc:2726] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 07:03:51.620973 16702 raft_consensus.cc:686] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 LEADER]: Becoming Leader. State: Replica: 3ab833e4a1114741a0f1fd146c8efb6d, State: Running, Role: LEADER
I20260620 07:03:51.621601 16702 consensus_queue.cc:237] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } }
I20260620 07:03:51.626055 16230 catalog_manager.cc:5495] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab833e4a1114741a0f1fd146c8efb6d (127.11.81.194). New cstate: current_term: 1 leader_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab833e4a1114741a0f1fd146c8efb6d" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36917 } health_report { overall_health: HEALTHY } } }
I20260620 07:03:51.746515 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.743988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.11.81.194:36917)
0620 07:03:51.744141 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.744146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.744663 (+ 517us) server_negotiation.cc:407] Connection header received
0620 07:03:51.744871 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.744875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.744957 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.745042 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.746013 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.746017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.746020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.746037 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.746072 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.746085 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.746224 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.746261 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.746310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:03:51.754410 16632 raft_consensus.cc:1261] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 1 FOLLOWER]: Refusing update from remote peer 3ab833e4a1114741a0f1fd146c8efb6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:51.754470 16371 raft_consensus.cc:1261] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 1 FOLLOWER]: Refusing update from remote peer 3ab833e4a1114741a0f1fd146c8efb6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:51.755218 16710 consensus_queue.cc:1035] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f20e25b4bfe142fd98d1e751fd32951a" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:51.755579 16702 consensus_queue.cc:1035] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9713c16d014ed49e7e3ef94740ee5a" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:51.766731 16714 mvcc.cc:205] Tried to move back new op lower bound from 7298822274056773632 to 7298822273525899264. Current Snapshot: MvccSnapshot[applied={T|T < 7298822274056773632}]
I20260620 07:03:51.781502 16229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:51.781805 16229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:03:51.785645 16229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43e691e0c7164e18a71d61cc33862034: Sending DeleteTablet for 3 replicas of tablet cfdd05bfb9ae4579aa719b8d2cbf83de
I20260620 07:03:51.786887 16351 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:51 UTC) from {username='slave'} at 127.0.0.1:58996
I20260620 07:03:51.786916 16482 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:51 UTC) from {username='slave'} at 127.0.0.1:59182
I20260620 07:03:51.787039 16612 tablet_service.cc:1514] Processing DeleteTablet for tablet cfdd05bfb9ae4579aa719b8d2cbf83de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:03:51 UTC) from {username='slave'} at 127.0.0.1:40408
I20260620 07:03:51.789129 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16289
I20260620 07:03:51.800690 16723 tablet_replica.cc:330] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: stopping tablet replica
I20260620 07:03:51.801241 16723 raft_consensus.cc:2229] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:03:51.802104 16723 raft_consensus.cc:2258] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:51.802971 16724 tablet_replica.cc:330] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: stopping tablet replica
I20260620 07:03:51.803475 16724 raft_consensus.cc:2229] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:51.804023 16724 raft_consensus.cc:2258] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:51.806272 16724 ts_tablet_manager.cc:1905] T cfdd05bfb9ae4579aa719b8d2cbf83de P f20e25b4bfe142fd98d1e751fd32951a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:51.809595 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.790844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.11.81.193:34389)
0620 07:03:51.791029 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.791035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.806194 (+ 15159us) server_negotiation.cc:407] Connection header received
0620 07:03:51.806758 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.806764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.806844 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.806917 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.808858 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.808863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.808867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.808888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.808927 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.808943 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.809032 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.809268 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.809331 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:03:51.810808 16411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:51.812623 16723 ts_tablet_manager.cc:1905] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:51.817198 16725 tablet_replica.cc:330] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: stopping tablet replica
I20260620 07:03:51.819144 16725 raft_consensus.cc:2229] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:03:51.820549 16725 raft_consensus.cc:2258] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:03:51.822614 16725 ts_tablet_manager.cc:1905] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:03:51.827508 16723 ts_tablet_manager.cc:1918] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:51.827828 16723 log.cc:1196] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/cfdd05bfb9ae4579aa719b8d2cbf83de
I20260620 07:03:51.828512 16723 ts_tablet_manager.cc:1939] T cfdd05bfb9ae4579aa719b8d2cbf83de P 3ab833e4a1114741a0f1fd146c8efb6d: Deleting consensus metadata
I20260620 07:03:51.829841 16725 ts_tablet_manager.cc:1918] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:03:51.830087 16725 log.cc:1196] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/cfdd05bfb9ae4579aa719b8d2cbf83de
I20260620 07:03:51.830624 16725 ts_tablet_manager.cc:1939] T cfdd05bfb9ae4579aa719b8d2cbf83de P 5a9713c16d014ed49e7e3ef94740ee5a: Deleting consensus metadata
I20260620 07:03:51.830632 16215 catalog_manager.cc:4841] TS 3ab833e4a1114741a0f1fd146c8efb6d (127.11.81.194:36917): tablet cfdd05bfb9ae4579aa719b8d2cbf83de (table test-table [id=2af2404abd124d4883856e022616f5d3]) successfully deleted
I20260620 07:03:51.832530 16214 catalog_manager.cc:4841] TS 5a9713c16d014ed49e7e3ef94740ee5a (127.11.81.195:43289): tablet cfdd05bfb9ae4579aa719b8d2cbf83de (table test-table [id=2af2404abd124d4883856e022616f5d3]) successfully deleted
I20260620 07:03:51.908735 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16289
W20260620 07:03:51.929260 16217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:34389 (error 108)
I20260620 07:03:51.929958 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16420
I20260620 07:03:51.931659 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.930594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34416)
0620 07:03:51.931299 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.931439 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":473,"threads_started":1}
W20260620 07:03:51.931944 16217 catalog_manager.cc:4584] TS f20e25b4bfe142fd98d1e751fd32951a (127.11.81.193:34389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfdd05bfb9ae4579aa719b8d2cbf83de: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
I20260620 07:03:51.933048 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.930779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.11.81.194:36917)
0620 07:03:51.930954 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:51.930959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:51.931208 (+ 249us) server_negotiation.cc:407] Connection header received
0620 07:03:51.931274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:51.931279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:51.931362 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:51.931461 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:51.932406 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:51.932412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:51.932418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:51.932438 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:51.932476 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:51.932491 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:51.932624 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:51.932670 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 07:03:51.932726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:03:51.933761 16542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:51.985482 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.985027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34424)
0620 07:03:51.985248 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.985339 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 07:03:51.986513 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:51.986112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34438)
0620 07:03:51.986290 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:51.986382 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:03:52.042680 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16420
I20260620 07:03:52.046959 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.046469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34454)
0620 07:03:52.046677 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:52.046786 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:03:52.048055 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.047649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34456)
0620 07:03:52.047794 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:52.047880 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:03:52.058169 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16550
I20260620 07:03:52.060760 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.058966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.11.81.195:43289)
0620 07:03:52.059147 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:52.059166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:52.059184 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:52.059243 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:52.059246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:52.059360 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:52.059441 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:52.060227 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:52.060231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:52.060234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:52.060249 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:52.060281 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:52.060293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:52.060390 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:52.060459 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 07:03:52.060504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:03:52.061465 16672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:52.129195 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.128688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34458)
0620 07:03:52.128958 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:52.129046 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 07:03:52.130270 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.129831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:34389 (local address 127.0.0.1:34466)
0620 07:03:52.130042 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:52.130115 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 07:03:52.149037 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16550
I20260620 07:03:52.161726 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16199
I20260620 07:03:52.162936 16260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:52.267592 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16199
2026-06-20T07:03:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3028 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 07:03:52.304704 11591 mini_oidc.cc:152] Starting JWKS server
I20260620 07:03:52.307243 11591 webserver.cc:460] Webserver started at https://127.0.0.1:38523/ using document root <none> and password file <none>
I20260620 07:03:52.307580 11591 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:03:52.308218 11591 webserver.cc:460] Webserver started at http://127.0.0.1:46113/ using document root <none> and password file <none>
2026-06-20T07:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:52Z Disabled control of system clock
I20260620 07:03:52.333546 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:37303
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:37303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38523/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/testchainca.cert with env {}
W20260620 07:03:52.655262 16738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:52.655655 16738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:52.655751 16738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:52.663112 16738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:52.663235 16738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:52.663272 16738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:52.663376 16738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:52.663431 16738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:03:52.663493 16738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:03:52.663532 16738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38523/jwks.json
I20260620 07:03:52.673874 16738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:37303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38523/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:52.675175 16738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:52.676522 16738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:52.688937 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:52.689005 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:52.688969 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:52.689729 16738 server_base.cc:1034] running on GCE node
I20260620 07:03:52.690590 16738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:52.692435 16738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:52.693710 16738 hybrid_clock.cc:648] HybridClock initialized: now 1781939032693671 us; error 39 us; skew 500 ppm
I20260620 07:03:52.694088 16738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:52.696583 16738 webserver.cc:460] Webserver started at http://127.11.81.254:41547/ using document root <none> and password file <none>
I20260620 07:03:52.697326 16738 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:52.697443 16738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:52.697757 16738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:52.700870 16738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "e473584b83fa474b9ea9217eb1ba359b"
format_stamp: "Formatted at 2026-06-20 07:03:52 on dist-test-slave-wvz3"
server_key: "8ce08f4bd3eaa9198b9c43c7d1081ea7"
server_key_iv: "047dbf0c672ae110337c3699ee0a2d8b"
server_key_version: "encryptionkey@0"
I20260620 07:03:52.701709 16738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "e473584b83fa474b9ea9217eb1ba359b"
format_stamp: "Formatted at 2026-06-20 07:03:52 on dist-test-slave-wvz3"
server_key: "8ce08f4bd3eaa9198b9c43c7d1081ea7"
server_key_iv: "047dbf0c672ae110337c3699ee0a2d8b"
server_key_version: "encryptionkey@0"
I20260620 07:03:52.707230 16738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:03:52.710984 16752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:52.712459 16738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:52.712646 16738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "e473584b83fa474b9ea9217eb1ba359b"
format_stamp: "Formatted at 2026-06-20 07:03:52 on dist-test-slave-wvz3"
server_key: "8ce08f4bd3eaa9198b9c43c7d1081ea7"
server_key_iv: "047dbf0c672ae110337c3699ee0a2d8b"
server_key_version: "encryptionkey@0"
I20260620 07:03:52.712853 16738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:52.729403 16738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:52.730293 16738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:52.730589 16738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:52.747852 16738 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:37303
I20260620 07:03:52.747876 16803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:37303 every 8 connection(s)
I20260620 07:03:52.749184 16738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:52.751909 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16738
I20260620 07:03:52.752126 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:52.752859 11591 external_mini_cluster.cc:1428] Setting key a6caa561f9c08333a1b669edfb22348d
I20260620 07:03:52.753535 16804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:52.768508 16804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b: Bootstrap starting.
I20260620 07:03:52.768532 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.754795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.11.81.254:37303)
0620 07:03:52.755898 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:52.755913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:52.755966 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:03:52.758301 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:52.758334 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:52.758995 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:52.759488 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:52.760077 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.762021 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.763434 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.764445 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.764677 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:52.765751 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:52.765761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:52.765783 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:52.766507 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:52.766603 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:52.766638 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:52.766741 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:52.766931 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 07:03:52.767268 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":337,"threads_started":1}
I20260620 07:03:52.772099 16804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:52.772657 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:52.773311 16804 log.cc:824] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:52.776448 16804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b: No bootstrap required, opened a new log
I20260620 07:03:52.783677 16804 raft_consensus.cc:348] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } }
I20260620 07:03:52.784044 16804 raft_consensus.cc:374] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:52.784114 16804 raft_consensus.cc:729] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e473584b83fa474b9ea9217eb1ba359b, State: Initialized, Role: FOLLOWER
I20260620 07:03:52.784677 16804 consensus_queue.cc:260] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } }
I20260620 07:03:52.784883 16804 raft_consensus.cc:388] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:52.784976 16804 raft_consensus.cc:482] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:52.785094 16804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:52.787477 16804 raft_consensus.cc:504] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } }
I20260620 07:03:52.787983 16804 leader_election.cc:302] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e473584b83fa474b9ea9217eb1ba359b; no voters:
I20260620 07:03:52.788581 16804 leader_election.cc:288] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:52.788756 16809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:52.789392 16809 raft_consensus.cc:686] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [term 1 LEADER]: Becoming Leader. State: Replica: e473584b83fa474b9ea9217eb1ba359b, State: Running, Role: LEADER
I20260620 07:03:52.789928 16809 consensus_queue.cc:237] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } }
I20260620 07:03:52.790601 16804 sys_catalog.cc:564] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:52.792735 16811 sys_catalog.cc:455] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e473584b83fa474b9ea9217eb1ba359b. Latest consensus state: current_term: 1 leader_uuid: "e473584b83fa474b9ea9217eb1ba359b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } } }
I20260620 07:03:52.793207 16811 sys_catalog.cc:458] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:52.792783 16810 sys_catalog.cc:455] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e473584b83fa474b9ea9217eb1ba359b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e473584b83fa474b9ea9217eb1ba359b" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 37303 } } }
I20260620 07:03:52.793433 16810 sys_catalog.cc:458] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:52.795795 16818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:52.799991 16818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:52.808490 16818 catalog_manager.cc:1347] Generated new cluster ID: 35f7df259198458490afa567123a58f5
I20260620 07:03:52.808612 16818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:52.820842 16818 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:52.822691 16818 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:52.831269 16818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e473584b83fa474b9ea9217eb1ba359b: Generated new TSK 0
I20260620 07:03:52.832234 16818 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:03:52.877562 16834 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 07:03:52.877786 16834 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:03:52.878909 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:52.856770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.11.81.254:37303)
0620 07:03:52.856982 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:52.856988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:52.857446 (+ 458us) server_negotiation.cc:407] Connection header received
0620 07:03:52.857784 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:52.857790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:52.857895 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:52.858057 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:03:52.858632 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.859847 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.861433 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.862126 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.862245 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:52.862836 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:03:52.878227 (+ 15391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:03:52.878600 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 07:03:52.879026 16807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 07:03:52.880293 16833 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:52.856646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:37303 (local address 127.0.0.1:50500)
0620 07:03:52.857334 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:52.857351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:52.857505 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:52.858121 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:52.858125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:52.858143 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:03:52.858455 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:52.858464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.860027 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.860032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:52.861279 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:52.861293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:52.862302 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:52.862307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:52.862446 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:52.862528 (+ 82us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:03:52.878501 (+ 15973us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:03:52.878835 (+ 334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.81.254:37303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":334,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.81.254:37303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38523/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 07:03:52.881901 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16738
I20260620 07:03:52.883015 16799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:52.976912 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16738
2026-06-20T07:03:53Z chronyd exiting
I20260620 07:03:53.024135 11591 test_util.cc:175] -----------------------------------------------
I20260620 07:03:53.024292 11591 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938992575046-11591-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 07:03:53.028180 11591 mini_oidc.cc:152] Starting JWKS server
I20260620 07:03:53.030076 11591 webserver.cc:460] Webserver started at https://127.0.0.1:34845/ using document root <none> and password file <none>
I20260620 07:03:53.030294 11591 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:03:53.030993 11591 webserver.cc:460] Webserver started at http://127.0.0.1:37455/ using document root <none> and password file <none>
2026-06-20T07:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:53Z Disabled control of system clock
I20260620 07:03:53.057651 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:45971
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:45971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/testchainca.cert with env {}
W20260620 07:03:53.384384 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:53.384757 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:53.384861 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:53.392179 16841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:53.392289 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:53.392325 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:53.392379 16841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:53.392421 16841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:03:53.392479 16841 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:03:53.392517 16841 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34845/jwks.json
I20260620 07:03:53.402568 16841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:45971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:53.403923 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:53.405192 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:53.417054 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:53.416965 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:53.416997 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:53.417688 16841 server_base.cc:1034] running on GCE node
I20260620 07:03:53.418449 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:53.419926 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:53.421123 16841 hybrid_clock.cc:648] HybridClock initialized: now 1781939033421078 us; error 33 us; skew 500 ppm
I20260620 07:03:53.421464 16841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:53.423708 16841 webserver.cc:460] Webserver started at http://127.11.81.254:41965/ using document root <none> and password file <none>
I20260620 07:03:53.424372 16841 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:53.424460 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:53.424726 16841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:53.427611 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "d7fb99ec015d4a97b3c4b0832771133d"
format_stamp: "Formatted at 2026-06-20 07:03:53 on dist-test-slave-wvz3"
server_key: "36a620a6c498380f4679ecc22c020988"
server_key_iv: "2c6e1962cc4b39389a76a95fadd0f418"
server_key_version: "encryptionkey@0"
I20260620 07:03:53.428383 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "d7fb99ec015d4a97b3c4b0832771133d"
format_stamp: "Formatted at 2026-06-20 07:03:53 on dist-test-slave-wvz3"
server_key: "36a620a6c498380f4679ecc22c020988"
server_key_iv: "2c6e1962cc4b39389a76a95fadd0f418"
server_key_version: "encryptionkey@0"
I20260620 07:03:53.433580 16841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:03:53.436988 16855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:53.438473 16841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:53.438628 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "d7fb99ec015d4a97b3c4b0832771133d"
format_stamp: "Formatted at 2026-06-20 07:03:53 on dist-test-slave-wvz3"
server_key: "36a620a6c498380f4679ecc22c020988"
server_key_iv: "2c6e1962cc4b39389a76a95fadd0f418"
server_key_version: "encryptionkey@0"
I20260620 07:03:53.438810 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:53.471457 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:53.472306 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:53.472573 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:53.493718 16841 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:45971
I20260620 07:03:53.493744 16906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:45971 every 8 connection(s)
I20260620 07:03:53.495129 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:53.497898 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16841
I20260620 07:03:53.498103 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:53.498888 11591 external_mini_cluster.cc:1428] Setting key 1c8c0a8ceeb212256c53c6e8062823a2
I20260620 07:03:53.499651 16907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:53.514950 16907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d: Bootstrap starting.
I20260620 07:03:53.515256 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:53.501059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.11.81.254:45971)
0620 07:03:53.502259 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:53.502279 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:53.502336 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:03:53.504476 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:53.504514 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:53.505206 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:53.505695 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:53.506309 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.508320 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.509712 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.510762 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.510962 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:53.512184 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:53.512194 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:53.512215 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:53.512986 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:53.513102 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:53.513129 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:53.513273 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:53.513513 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 07:03:53.513884 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":372,"threads_started":1}
I20260620 07:03:53.518720 16907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:53.519785 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:53.519969 16907 log.cc:824] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:53.523121 16907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d: No bootstrap required, opened a new log
I20260620 07:03:53.530896 16907 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } }
I20260620 07:03:53.531360 16907 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:53.531462 16907 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7fb99ec015d4a97b3c4b0832771133d, State: Initialized, Role: FOLLOWER
I20260620 07:03:53.532058 16907 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } }
I20260620 07:03:53.532282 16907 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:53.532416 16907 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:53.532588 16907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:53.534792 16907 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } }
I20260620 07:03:53.535398 16907 leader_election.cc:302] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7fb99ec015d4a97b3c4b0832771133d; no voters:
I20260620 07:03:53.536093 16907 leader_election.cc:288] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:53.536198 16912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:53.536849 16912 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [term 1 LEADER]: Becoming Leader. State: Replica: d7fb99ec015d4a97b3c4b0832771133d, State: Running, Role: LEADER
I20260620 07:03:53.537343 16912 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } }
I20260620 07:03:53.538165 16907 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:53.540163 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7fb99ec015d4a97b3c4b0832771133d. Latest consensus state: current_term: 1 leader_uuid: "d7fb99ec015d4a97b3c4b0832771133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } } }
I20260620 07:03:53.540584 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:53.542395 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7fb99ec015d4a97b3c4b0832771133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7fb99ec015d4a97b3c4b0832771133d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 45971 } } }
I20260620 07:03:53.542734 16920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:53.542790 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:53.548197 16920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:53.558416 16920 catalog_manager.cc:1347] Generated new cluster ID: 1f59c07bf7de4615912c92bb0de008ea
I20260620 07:03:53.558627 16920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:53.586697 16920 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:53.588297 16920 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:53.603058 16920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7fb99ec015d4a97b3c4b0832771133d: Generated new TSK 0
I20260620 07:03:53.604211 16920 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:03:53.649672 16937 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 07:03:53.649930 16937 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:03:53.651049 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:53.629819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.11.81.254:45971)
0620 07:03:53.630020 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:53.630026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:53.630348 (+ 322us) server_negotiation.cc:407] Connection header received
0620 07:03:53.630611 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:53.630617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:53.630718 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:53.630898 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:03:53.631427 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.632687 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.634194 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.634872 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.635001 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:53.635592 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:03:53.650301 (+ 14709us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:03:53.650722 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 07:03:53.651185 16910 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 07:03:53.652257 16936 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:53.629654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:45971 (local address 127.0.0.1:44946)
0620 07:03:53.630239 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:53.630256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:53.630391 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:53.630928 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:53.630933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:53.630944 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:03:53.631219 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:53.631228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.632849 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.632854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:53.634040 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:53.634052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:53.635075 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:53.635080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:53.635226 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:53.635264 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:03:53.650725 (+ 15461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:03:53.651709 (+ 984us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.81.254:45971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":313,"threads_started":1}
I20260620 07:03:53.653623 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16841
I20260620 07:03:53.654889 16902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:53.749929 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16841
2026-06-20T07:03:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 07:03:53.802011 11591 mini_oidc.cc:152] Starting JWKS server
I20260620 07:03:53.803970 11591 webserver.cc:460] Webserver started at https://127.0.0.1:39915/ using document root <none> and password file <none>
I20260620 07:03:53.804212 11591 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:03:53.804850 11591 webserver.cc:460] Webserver started at http://127.0.0.1:46275/ using document root <none> and password file <none>
2026-06-20T07:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:53Z Disabled control of system clock
I20260620 07:03:53.830844 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38203
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:42047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:38203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39915/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:03:54.154825 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:54.155126 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:54.155236 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:54.162577 16944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:54.162700 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:54.162740 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:54.162822 16944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:54.162876 16944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:03:54.162938 16944 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:03:54.162992 16944 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39915/jwks.json
I20260620 07:03:54.172335 16944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:38203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39915/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:54.173648 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:54.175411 16944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:54.187013 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:54.186947 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:54.187049 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:54.188120 16944 server_base.cc:1034] running on GCE node
I20260620 07:03:54.189014 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:54.190682 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:54.191907 16944 hybrid_clock.cc:648] HybridClock initialized: now 1781939034191855 us; error 35 us; skew 500 ppm
I20260620 07:03:54.192289 16944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:54.194762 16944 webserver.cc:460] Webserver started at http://127.11.81.254:35309/ using document root <none> and password file <none>
I20260620 07:03:54.195529 16944 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:54.195636 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:54.195951 16944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:54.199156 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "93425136af73467a8b323d9c52785e66"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "d8d0e712f033a2aacda0818ec6375843"
server_key_iv: "fedfd63172eb4137f0e61de69789c3dd"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.200060 16944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "93425136af73467a8b323d9c52785e66"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "d8d0e712f033a2aacda0818ec6375843"
server_key_iv: "fedfd63172eb4137f0e61de69789c3dd"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.205569 16944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:03:54.209350 16958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:54.210968 16944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:03:54.211148 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "93425136af73467a8b323d9c52785e66"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "d8d0e712f033a2aacda0818ec6375843"
server_key_iv: "fedfd63172eb4137f0e61de69789c3dd"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.211385 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:54.231974 16944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:54.232839 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:54.233141 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:54.251753 16944 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:38203
I20260620 07:03:54.251763 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:38203 every 8 connection(s)
I20260620 07:03:54.253199 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:54.257122 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:54.259969 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 16944
I20260620 07:03:54.260180 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:54.260792 11591 external_mini_cluster.cc:1428] Setting key f2facd38da198880e78aaba4ec1d7269
I20260620 07:03:54.268821 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66: Bootstrap starting.
I20260620 07:03:54.271971 17010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:54.273046 17010 log.cc:824] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:54.274111 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:54.262778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.11.81.254:38203)
0620 07:03:54.263870 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:54.263882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:54.263928 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:03:54.265539 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:54.265562 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:54.266063 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:54.266459 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:54.267041 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.268313 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.269611 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.270295 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.270470 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:54.271613 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:54.271620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:54.271634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:54.272183 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:54.272280 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:54.272301 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:54.272409 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:54.272620 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 07:03:54.272914 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":353,"threads_started":1}
I20260620 07:03:54.275928 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66: No bootstrap required, opened a new log
I20260620 07:03:54.277828 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:54.284039 17010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } }
I20260620 07:03:54.284377 17010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:54.284446 17010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93425136af73467a8b323d9c52785e66, State: Initialized, Role: FOLLOWER
I20260620 07:03:54.284935 17010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } }
I20260620 07:03:54.285123 17010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:54.285252 17010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:54.285406 17010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:54.287649 17010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } }
I20260620 07:03:54.288121 17010 leader_election.cc:302] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93425136af73467a8b323d9c52785e66; no voters:
I20260620 07:03:54.288671 17010 leader_election.cc:288] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:54.288805 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:54.289789 17015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [term 1 LEADER]: Becoming Leader. State: Replica: 93425136af73467a8b323d9c52785e66, State: Running, Role: LEADER
I20260620 07:03:54.290295 17015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } }
I20260620 07:03:54.290735 17010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:54.293016 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93425136af73467a8b323d9c52785e66. Latest consensus state: current_term: 1 leader_uuid: "93425136af73467a8b323d9c52785e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } } }
I20260620 07:03:54.293448 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:54.294023 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93425136af73467a8b323d9c52785e66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93425136af73467a8b323d9c52785e66" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38203 } } }
I20260620 07:03:54.294314 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:54.295325 17024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:54.300773 17024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:54.309715 17024 catalog_manager.cc:1347] Generated new cluster ID: cc04d600749846769b41279cf8afbae7
I20260620 07:03:54.309845 17024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:54.330308 17024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:54.331863 17024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:54.346405 17024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93425136af73467a8b323d9c52785e66: Generated new TSK 0
I20260620 07:03:54.347304 17024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:03:54.372537 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:54.363892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.11.81.254:38203)
0620 07:03:54.364112 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:54.364119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:54.364579 (+ 460us) server_negotiation.cc:407] Connection header received
0620 07:03:54.364811 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:54.364815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:54.364901 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:54.365031 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:03:54.365588 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.366318 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.368426 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.368991 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.369101 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:54.369674 (+ 573us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:03:54.371894 (+ 2220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39915/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:03:54.372229 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39915/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:03:54.372654 17013 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39915/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 07:03:54.374702 17039 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:54.363856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38203 (local address 127.0.0.1:38438)
0620 07:03:54.364466 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:54.364487 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:54.364643 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:54.365064 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:54.365069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:54.365082 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:03:54.365406 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:54.365418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.366462 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.366468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:54.368224 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:54.368239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.369189 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.369196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:54.369383 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:54.369431 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:03:54.372119 (+ 2688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39915/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:03:54.374228 (+ 2109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.81.254:38203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39915/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":1785,"thread_start_us":290,"threads_started":1}
I20260620 07:03:54.375947 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 16944
I20260620 07:03:54.377202 17005 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 07:03:54.387207 17040 webserver.cc:533] Webserver: sslize failed:
E20260620 07:03:54.387372 17040 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:03:54.472386 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 16944
2026-06-20T07:03:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T07:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:54Z Disabled control of system clock
I20260620 07:03:54.549187 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36503
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:36503
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/testchainca.cert with env {}
W20260620 07:03:54.868417 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:54.868738 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:54.868824 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:54.876298 17045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:54.876422 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:54.876458 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:54.876509 17045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:54.876551 17045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:03:54.876609 17045 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:03:54.876647 17045 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 07:03:54.886806 17045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:36503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:54.888111 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:54.889487 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:54.901254 17051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:54.901278 17053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:54.901222 17050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:54.902400 17045 server_base.cc:1034] running on GCE node
I20260620 07:03:54.903271 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:54.904837 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:54.906044 17045 hybrid_clock.cc:648] HybridClock initialized: now 1781939034905990 us; error 52 us; skew 500 ppm
I20260620 07:03:54.906387 17045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:03:54.914562 17045 webserver.cc:460] Webserver started at http://127.11.81.254:41261/ using document root <none> and password file <none>
I20260620 07:03:54.915483 17045 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:54.915617 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:54.915971 17045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:54.919239 17045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "522cc00e0add4c23a51b49562da240e0"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "21985e1eb4a98987c4fdc31142a01e86"
server_key_iv: "9b669eb8b3776e7b0ce6a363ff634e9d"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.920204 17045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "522cc00e0add4c23a51b49562da240e0"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "21985e1eb4a98987c4fdc31142a01e86"
server_key_iv: "9b669eb8b3776e7b0ce6a363ff634e9d"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.926013 17045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:03:54.929790 17059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:54.931394 17045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:54.931609 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "522cc00e0add4c23a51b49562da240e0"
format_stamp: "Formatted at 2026-06-20 07:03:54 on dist-test-slave-wvz3"
server_key: "21985e1eb4a98987c4fdc31142a01e86"
server_key_iv: "9b669eb8b3776e7b0ce6a363ff634e9d"
server_key_version: "encryptionkey@0"
I20260620 07:03:54.931823 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:54.950691 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:54.951588 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:54.951859 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:54.969798 17045 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:36503
I20260620 07:03:54.969825 17110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:36503 every 8 connection(s)
I20260620 07:03:54.971120 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:54.974938 17111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:54.978055 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17045
I20260620 07:03:54.978214 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:54.978852 11591 external_mini_cluster.cc:1428] Setting key 0bb274349e83a3adeed7e93b688a34ac
I20260620 07:03:54.988494 17111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0: Bootstrap starting.
I20260620 07:03:54.992259 17111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:54.993628 17111 log.cc:824] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:54.994388 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:54.980817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.11.81.254:36503)
0620 07:03:54.981900 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:54.981915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:54.981966 (+ 51us) server_negotiation.cc:407] Connection header received
0620 07:03:54.983995 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:54.984027 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:54.984724 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:54.985199 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:54.985781 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.987715 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.989116 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:54.990145 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:54.990328 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:54.991404 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:54.991415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:54.991433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:03:54.992168 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:54.992254 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:03:54.992279 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:54.992410 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:54.992689 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 07:03:54.993046 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":314,"threads_started":1}
I20260620 07:03:54.997154 17111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0: No bootstrap required, opened a new log
I20260620 07:03:54.998723 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:55.005960 17111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } }
I20260620 07:03:55.006397 17111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:55.006501 17111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522cc00e0add4c23a51b49562da240e0, State: Initialized, Role: FOLLOWER
I20260620 07:03:55.007112 17111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } }
I20260620 07:03:55.007367 17111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:55.007498 17111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:55.007659 17111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:55.010288 17111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } }
I20260620 07:03:55.010835 17111 leader_election.cc:302] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522cc00e0add4c23a51b49562da240e0; no voters:
I20260620 07:03:55.011479 17111 leader_election.cc:288] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:55.011643 17116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:55.012333 17116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [term 1 LEADER]: Becoming Leader. State: Replica: 522cc00e0add4c23a51b49562da240e0, State: Running, Role: LEADER
I20260620 07:03:55.012872 17116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } }
I20260620 07:03:55.013576 17111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:55.016067 17118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522cc00e0add4c23a51b49562da240e0. Latest consensus state: current_term: 1 leader_uuid: "522cc00e0add4c23a51b49562da240e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } } }
I20260620 07:03:55.016040 17117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522cc00e0add4c23a51b49562da240e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522cc00e0add4c23a51b49562da240e0" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36503 } } }
I20260620 07:03:55.016482 17118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:55.016533 17117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:55.018254 17124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:55.023562 17124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:55.032953 17124 catalog_manager.cc:1347] Generated new cluster ID: 7c299ebc17204fb8bf8d3c6b4aab0067
I20260620 07:03:55.033085 17124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:55.051421 17124 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:55.052742 17124 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:55.059491 17124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 522cc00e0add4c23a51b49562da240e0: Generated new TSK 0
I20260620 07:03:55.060392 17124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:03:55.102638 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:55.076632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.11.81.254:36503)
0620 07:03:55.076797 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:55.076803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:55.077256 (+ 453us) server_negotiation.cc:407] Connection header received
0620 07:03:55.077533 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:55.077538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:55.077617 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:55.077750 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:03:55.078284 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:55.079041 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:55.081124 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:55.081618 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:55.081721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:55.082329 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:03:55.101954 (+ 19625us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:03:55.102302 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48856: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 07:03:55.102897 17114 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48856: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 07:03:55.103086 17140 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:55.076455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36503 (local address 127.0.0.1:48856)
0620 07:03:55.077134 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:55.077155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:55.077320 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:55.077792 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:55.077797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:55.077810 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:03:55.078110 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:55.078122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:55.079191 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:55.079197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:55.080957 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:55.080974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:55.081802 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:55.081809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:55.082010 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:55.082064 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:03:55.102259 (+ 20195us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:03:55.102583 (+ 324us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.81.254:36503: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":337,"threads_started":1}
I20260620 07:03:55.104269 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17045
I20260620 07:03:55.105629 17106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:55.199524 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17045
2026-06-20T07:03:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (706 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1386 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:57 dist-test-slave-wvz3 krb5kdc[17156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:57 dist-test-slave-wvz3 krb5kdc[17156](info): set up 2 sockets
Jun 20 07:03:57 dist-test-slave-wvz3 krb5kdc[17156](info): commencing operation
krb5kdc: starting...
W20260620 07:03:58.003692 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:03:58 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:03:58.210834 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38389
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:38389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:58.524859 17169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:58.525166 17169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:58.525277 17169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:58.532557 17169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:58.532671 17169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:58.532748 17169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:58.532855 17169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:58.532909 17169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:58.543239 17169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:38389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:58.544592 17169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:58.545881 17169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:58.557276 17174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:58.557235 17175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:58.557207 17177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:58.558101 17169 server_base.cc:1034] running on GCE node
I20260620 07:03:58.558885 17169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:58.560364 17169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:58.561570 17169 hybrid_clock.cc:648] HybridClock initialized: now 1781939038561521 us; error 41 us; skew 500 ppm
I20260620 07:03:58.561906 17169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:58 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939038, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:58.564990 17169 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:03:58.566536 17169 webserver.cc:460] Webserver started at http://127.11.81.254:42601/ using document root <none> and password file <none>
I20260620 07:03:58.567221 17169 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:58.567363 17169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:58.567727 17169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:58.570901 17169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "25d6b29e2e494d838fd541f824d225d6"
format_stamp: "Formatted at 2026-06-20 07:03:58 on dist-test-slave-wvz3"
server_key: "b0f3c010ee387a60c2454c7f4ba18ed6"
server_key_iv: "4834121b2e23d9b71069bc7aadaac960"
server_key_version: "encryptionkey@0"
I20260620 07:03:58.571725 17169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "25d6b29e2e494d838fd541f824d225d6"
format_stamp: "Formatted at 2026-06-20 07:03:58 on dist-test-slave-wvz3"
server_key: "b0f3c010ee387a60c2454c7f4ba18ed6"
server_key_iv: "4834121b2e23d9b71069bc7aadaac960"
server_key_version: "encryptionkey@0"
I20260620 07:03:58.576903 17169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:58.580417 17184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:58.581727 17169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:03:58.581887 17169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "25d6b29e2e494d838fd541f824d225d6"
format_stamp: "Formatted at 2026-06-20 07:03:58 on dist-test-slave-wvz3"
server_key: "b0f3c010ee387a60c2454c7f4ba18ed6"
server_key_iv: "4834121b2e23d9b71069bc7aadaac960"
server_key_version: "encryptionkey@0"
I20260620 07:03:58.582069 17169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:58.599057 17169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:58.603044 17169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:58.603339 17169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:58.620345 17169 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:38389
I20260620 07:03:58.620364 17235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:38389 every 8 connection(s)
I20260620 07:03:58.621680 17169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:58.625289 17236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:58.629482 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17169
I20260620 07:03:58.629691 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:03:58.630460 11591 external_mini_cluster.cc:1428] Setting key 9ad9ea3ac412504ae86f6655618ba4fc
I20260620 07:03:58.636797 17236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6: Bootstrap starting.
I20260620 07:03:58.640298 17236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:58.641433 17236 log.cc:824] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:58.644709 17236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6: No bootstrap required, opened a new log
Jun 20 07:03:58 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:58.654832 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:58.632416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.11.81.254:38389)
0620 07:03:58.633172 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:58.633184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:58.633225 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:03:58.635349 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:58.635375 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:58.635833 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:58.636172 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:58.637831 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:58.639155 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:58.640858 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:58.641533 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:58.641680 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:58.646164 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:58.646175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:58.646208 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:58.646255 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:58.650178 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:58.650882 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:58.650894 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:58.650903 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:58.651107 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:58.651469 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:58.651473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:58.651476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:58.652283 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:58.652612 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:58.652933 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 07:03:58.653415 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":353,"threads_started":1}
I20260620 07:03:58.654955 17236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } }
I20260620 07:03:58.655462 17236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:03:58.655572 17236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d6b29e2e494d838fd541f824d225d6, State: Initialized, Role: FOLLOWER
I20260620 07:03:58.656200 17236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } }
I20260620 07:03:58.656443 17236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:03:58.656577 17236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:03:58.656756 17236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:58.658957 17236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } }
I20260620 07:03:58.659476 17236 leader_election.cc:302] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d6b29e2e494d838fd541f824d225d6; no voters:
I20260620 07:03:58.660063 17236 leader_election.cc:288] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:03:58.660241 17241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:58.660887 17241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [term 1 LEADER]: Becoming Leader. State: Replica: 25d6b29e2e494d838fd541f824d225d6, State: Running, Role: LEADER
I20260620 07:03:58.661371 17241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } }
I20260620 07:03:58.661926 17236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:58.663954 17242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d6b29e2e494d838fd541f824d225d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } } }
I20260620 07:03:58.664346 17242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:58.664793 17243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d6b29e2e494d838fd541f824d225d6. Latest consensus state: current_term: 1 leader_uuid: "25d6b29e2e494d838fd541f824d225d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d6b29e2e494d838fd541f824d225d6" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38389 } } }
I20260620 07:03:58.665099 17243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:58.665302 17247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:58.672147 17247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:58.681771 17259 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:03:58.683422 17247 catalog_manager.cc:1347] Generated new cluster ID: f07aacae91ff49f59d990404506c3b1b
I20260620 07:03:58.683540 17247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:58.704180 17247 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:58.706141 17247 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:58.714126 17247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d6b29e2e494d838fd541f824d225d6: Generated new TSK 0
I20260620 07:03:58.715224 17247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:03:58.763618 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:59.085536 17262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:59.085821 17262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:59.085918 17262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:59.093274 17262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:59.093389 17262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:59.093514 17262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:03:59.103185 17262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:59.104457 17262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:59.105683 17262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:59.118319 17268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:59.118386 17267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:59.118847 17270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:59.119084 17262 server_base.cc:1034] running on GCE node
I20260620 07:03:59.119787 17262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:59.120800 17262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:59.121986 17262 hybrid_clock.cc:648] HybridClock initialized: now 1781939039121932 us; error 59 us; skew 500 ppm
I20260620 07:03:59.122291 17262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:59 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:59.125295 17262 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:03:59.126746 17262 webserver.cc:460] Webserver started at http://127.11.81.193:37793/ using document root <none> and password file <none>
I20260620 07:03:59.127437 17262 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:59.127555 17262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:59.127849 17262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:59.130806 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "bd0fe909eace4e9384287182a3fd9a60"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "bf10eb62db3e4dc71f07f15d3241f895"
server_key_iv: "ebf8ebe52b8f5bf7e2c8b4304c2abbe1"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.131630 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "bd0fe909eace4e9384287182a3fd9a60"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "bf10eb62db3e4dc71f07f15d3241f895"
server_key_iv: "ebf8ebe52b8f5bf7e2c8b4304c2abbe1"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.136991 17262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:59.140676 17277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.142149 17262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:59.142331 17262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "bd0fe909eace4e9384287182a3fd9a60"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "bf10eb62db3e4dc71f07f15d3241f895"
server_key_iv: "ebf8ebe52b8f5bf7e2c8b4304c2abbe1"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.142580 17262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:59.175536 17262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:59.179841 17262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:59.180147 17262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:59.181169 17262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:59.182677 17262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:59.182765 17262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.182895 17262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:59.182964 17262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.215408 17262 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:34567
I20260620 07:03:59.215440 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:34567 every 8 connection(s)
I20260620 07:03:59.216892 17262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:03:59.223471 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17262
I20260620 07:03:59.223681 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:03:59.224370 11591 external_mini_cluster.cc:1428] Setting key 953ac148f11467ed352ddb77186bd2bf
Jun 20 07:03:59 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:59.237162 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:59.219596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:48837 (local address 127.11.81.254:38389)
0620 07:03:59.220032 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:59.220036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:59.221206 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 07:03:59.223015 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:59.223021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:59.223103 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:59.223216 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:59.224852 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.225684 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.227330 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.227727 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.227838 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:59.231344 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:59.231349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:59.231353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:59.231386 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:59.234196 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:59.234926 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:59.234932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:59.234936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:59.235050 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:59.235479 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:59.235484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:59.235488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:59.235748 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:59.235876 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:59.236576 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 07:03:59.236795 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
I20260620 07:03:59.238127 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:59.220258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38389 (local address 127.11.81.193:48837)
0620 07:03:59.221034 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:59.221080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:59.222739 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:59.223487 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:59.223496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:59.224076 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:59.224686 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:59.224706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.225853 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.225857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:59.227172 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:59.227185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.227943 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.227950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:59.228184 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:59.228856 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:59.228878 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:59.231102 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:59.234329 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:59.234336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:59.234350 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:59.234818 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:59.235151 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:59.235155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:59.235159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:59.235380 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:59.235865 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:59.235873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:59.236216 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 07:03:59.236453 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 07:03:59.236989 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":366,"threads_started":1}
I20260620 07:03:59.239722 17390 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38389
I20260620 07:03:59.240096 17390 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:59.240989 17390 heartbeater.cc:511] Master 127.11.81.254:38389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:03:59.243515 17201 ts_manager.cc:194] Registered new tserver with Master: bd0fe909eace4e9384287182a3fd9a60 (127.11.81.193:34567)
I20260620 07:03:59.244968 17201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:48837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:03:59.270229 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:59.588294 17396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:59.588585 17396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:59.588680 17396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:59.596127 17396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:59.596242 17396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:59.596355 17396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:03:59.606175 17396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:59.607483 17396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:59.608721 17396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:59.622112 17402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:59.622205 17404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:59.622709 17401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:59.623072 17396 server_base.cc:1034] running on GCE node
I20260620 07:03:59.623816 17396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:59.624958 17396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:59.626196 17396 hybrid_clock.cc:648] HybridClock initialized: now 1781939039626137 us; error 50 us; skew 500 ppm
I20260620 07:03:59.626554 17396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:59 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:59.630349 17396 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:03:59.632071 17396 webserver.cc:460] Webserver started at http://127.11.81.194:42497/ using document root <none> and password file <none>
I20260620 07:03:59.632941 17396 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:59.633076 17396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:59.633384 17396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:59.637305 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "4962d4a2502042a6a5ff0985cd859a99"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "d97eab66d060ac61f617d3409049bfd3"
server_key_iv: "ab530a72eb04371283899f495f3f4352"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.638273 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "4962d4a2502042a6a5ff0985cd859a99"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "d97eab66d060ac61f617d3409049bfd3"
server_key_iv: "ab530a72eb04371283899f495f3f4352"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.643944 17396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:59.647408 17411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.648821 17396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:59.648968 17396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "4962d4a2502042a6a5ff0985cd859a99"
format_stamp: "Formatted at 2026-06-20 07:03:59 on dist-test-slave-wvz3"
server_key: "d97eab66d060ac61f617d3409049bfd3"
server_key_iv: "ab530a72eb04371283899f495f3f4352"
server_key_version: "encryptionkey@0"
I20260620 07:03:59.649163 17396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:59.664321 17396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:59.668437 17396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:59.668711 17396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:59.669685 17396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:03:59.671017 17396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:03:59.671099 17396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.671214 17396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:03:59.671270 17396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:59.702764 17396 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:45699
I20260620 07:03:59.702801 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:45699 every 8 connection(s)
I20260620 07:03:59.704068 17396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:03:59.709636 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17396
I20260620 07:03:59.709816 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:03:59.710337 11591 external_mini_cluster.cc:1428] Setting key f354814cfa4a864bdc3df96aba6395f9
Jun 20 07:03:59 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939039, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:03:59.725106 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:59.706456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:33401 (local address 127.11.81.254:38389)
0620 07:03:59.706639 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:59.706644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:59.707878 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 07:03:59.709924 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:59.709930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:59.710020 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:59.710122 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:59.712091 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.712891 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.715128 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.715585 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.715686 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:59.719559 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:59.719564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:59.719567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:59.719600 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:59.722037 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:59.722788 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:59.722792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:59.722795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:59.722920 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:59.723406 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:59.723409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:59.723411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:59.723702 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:59.723830 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:59.724547 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 07:03:59.724776 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:03:59.726081 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:59.706940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38389 (local address 127.11.81.194:33401)
0620 07:03:59.707709 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:59.707747 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:59.709593 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:59.710419 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:59.710432 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:59.711121 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:59.711897 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:59.711923 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.713059 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.713065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:59.714974 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:59.714989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:59.715790 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:59.715796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:59.716079 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:59.716775 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:59.716805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:59.719337 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:59.722193 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:59.722204 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:59.722223 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:59.722640 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:59.723080 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:59.723084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:59.723086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:59.723269 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:59.723823 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:59.723832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:59.724187 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 07:03:59.724417 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 07:03:59.724946 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":324,"threads_started":1}
I20260620 07:03:59.727543 17524 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38389
I20260620 07:03:59.727839 17524 heartbeater.cc:464] Registering TS with master...
I20260620 07:03:59.728528 17524 heartbeater.cc:511] Master 127.11.81.254:38389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:03:59.729983 17201 ts_manager.cc:194] Registered new tserver with Master: 4962d4a2502042a6a5ff0985cd859a99 (127.11.81.194:45699)
I20260620 07:03:59.730805 17201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:33401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:03:59.755261 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:00.074676 17529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:00.074972 17529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:00.075115 17529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:00.082700 17529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:00.082831 17529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:00.082993 17529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:00.093292 17529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:00.094638 17529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:00.095953 17529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:00.108934 17535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:00.109066 17534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:00.108966 17537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:00.109789 17529 server_base.cc:1034] running on GCE node
I20260620 07:04:00.110461 17529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:00.111477 17529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:00.112715 17529 hybrid_clock.cc:648] HybridClock initialized: now 1781939040112646 us; error 52 us; skew 500 ppm
I20260620 07:04:00.113137 17529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.120534 17529 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:04:00.122288 17529 webserver.cc:460] Webserver started at http://127.11.81.195:44823/ using document root <none> and password file <none>
I20260620 07:04:00.123039 17529 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:00.123162 17529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:00.123523 17529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:00.126549 17529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "13f8825a50e9449380c037b81bd2e66b"
format_stamp: "Formatted at 2026-06-20 07:04:00 on dist-test-slave-wvz3"
server_key: "ddf6a49d4c64a9043c00ee080295f7bb"
server_key_iv: "cb55c31ec67bcaa957b8b1bd823beb62"
server_key_version: "encryptionkey@0"
I20260620 07:04:00.127389 17529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "13f8825a50e9449380c037b81bd2e66b"
format_stamp: "Formatted at 2026-06-20 07:04:00 on dist-test-slave-wvz3"
server_key: "ddf6a49d4c64a9043c00ee080295f7bb"
server_key_iv: "cb55c31ec67bcaa957b8b1bd823beb62"
server_key_version: "encryptionkey@0"
I20260620 07:04:00.132822 17529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:04:00.136413 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:00.137868 17529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:04:00.138027 17529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "13f8825a50e9449380c037b81bd2e66b"
format_stamp: "Formatted at 2026-06-20 07:04:00 on dist-test-slave-wvz3"
server_key: "ddf6a49d4c64a9043c00ee080295f7bb"
server_key_iv: "cb55c31ec67bcaa957b8b1bd823beb62"
server_key_version: "encryptionkey@0"
I20260620 07:04:00.138229 17529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:00.157524 17529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:00.161913 17529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:00.162197 17529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:00.163302 17529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:00.164721 17529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:00.164795 17529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:00.164916 17529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:00.164963 17529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:00.194097 17529 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:45717
I20260620 07:04:00.194176 17656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:45717 every 8 connection(s)
I20260620 07:04:00.195420 17529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:00.205010 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17529
I20260620 07:04:00.205207 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:00.205829 11591 external_mini_cluster.cc:1428] Setting key f7dc8eb7664e832e162ac42228bfdd91
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:00.216727 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.198044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:59645 (local address 127.11.81.254:38389)
0620 07:04:00.198248 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.198252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.199561 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 07:04:00.201500 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.201505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.201589 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.201698 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.203393 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.204267 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.206609 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.207188 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.207353 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.210931 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.210936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.210939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.210974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.213567 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.214342 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.214346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.214348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.214497 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.214912 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.214916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.214918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.215221 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.215395 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.216118 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.216354 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:04:00.217767 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.198608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38389 (local address 127.11.81.195:59645)
0620 07:04:00.199386 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:00.199427 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:00.201105 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:00.201919 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:00.201928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:00.202527 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:00.203154 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:00.203174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.204426 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.204430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:00.206420 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:00.206434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.207455 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.207463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:00.207749 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.208405 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:00.208434 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:00.210699 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:00.213723 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:00.213732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:00.213748 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:00.214210 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:00.214593 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:00.214598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:00.214601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:00.214790 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:00.215379 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:00.215388 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:00.215738 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 07:04:00.215995 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 07:04:00.216565 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":281,"threads_started":1}
I20260620 07:04:00.219228 17657 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38389
I20260620 07:04:00.219537 17657 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:00.220445 17657 heartbeater.cc:511] Master 127.11.81.254:38389 requested a full tablet report, sending...
I20260620 07:04:00.221932 17201 ts_manager.cc:194] Registered new tserver with Master: 13f8825a50e9449380c037b81bd2e66b (127.11.81.195:45717)
I20260620 07:04:00.222724 17201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:59645
I20260620 07:04:00.233965 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:04:00.248744 17390 heartbeater.cc:503] Master 127.11.81.254:38389 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/test-admin.keytab.
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.262887 11591 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 07:04:00.267838 17664 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:00.265549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.11.81.193:34567)
0620 07:04:00.266151 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.266159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.266186 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:00.266249 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.266255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.266472 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.266610 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.267461 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33846: BlockingRecv error: recv got EOF from 127.0.0.1:33846 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
W20260620 07:04:00.268309 11591 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc)
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.269866 11591 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.271227 11591 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:04:00.286363 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.272997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.11.81.193:34567)
0620 07:04:00.273187 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.273194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.273364 (+ 170us) server_negotiation.cc:407] Connection header received
0620 07:04:00.273532 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.273536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.273606 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.273690 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.274713 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.275736 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.276926 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.277486 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.277611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.281133 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.281145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.281160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.281194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.283776 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.284378 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.284385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.284387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.284503 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.284814 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.284817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.284819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.285199 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.285412 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.285810 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.286011 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:04:00.292404 17664 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:00.290652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.11.81.193:34567)
0620 07:04:00.290810 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.290814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.291119 (+ 305us) server_negotiation.cc:407] Connection header received
0620 07:04:00.291239 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.291243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.291352 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.291424 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.292156 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33870: BlockingRecv error: recv got EOF from 127.0.0.1:33870 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:04:00.292698 11591 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.294149 11591 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.295409 11591 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:04:00.309299 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.296763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.11.81.193:34567)
0620 07:04:00.296896 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.296900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.297356 (+ 456us) server_negotiation.cc:407] Connection header received
0620 07:04:00.297504 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.297508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.297566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.297655 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.298567 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.299250 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.300416 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.300846 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.300936 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.304508 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.304514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.304517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.304555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.307023 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.307687 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.307691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.307693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.307809 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.308130 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.308134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.308136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.308449 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.308668 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.308867 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.309052 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 07:04:00.313689 17664 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:00.312082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.11.81.193:34567)
0620 07:04:00.312279 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.312285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.312466 (+ 181us) server_negotiation.cc:407] Connection header received
0620 07:04:00.312615 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.312619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.312703 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.312777 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.313429 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33886: BlockingRecv error: recv got EOF from 127.0.0.1:33886 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 07:04:00.313797 11591 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938992575046-11591-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.315433 11591 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:00.316726 11591 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:04:00.330219 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.318015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.11.81.193:34567)
0620 07:04:00.318208 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.318213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.318371 (+ 158us) server_negotiation.cc:407] Connection header received
0620 07:04:00.318619 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.318623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.318703 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.318788 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.319748 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.320442 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.321666 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.322064 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.322153 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.325453 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.325458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.325461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.325496 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.328039 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.328667 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.328671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.328673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.328801 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.329065 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.329069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.329072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.329333 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.329562 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.329733 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.329890 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:04:00.331786 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17262
I20260620 07:04:00.343053 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.332423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.11.81.193:34567)
0620 07:04:00.332600 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.332605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.332880 (+ 275us) server_negotiation.cc:407] Connection header received
0620 07:04:00.333023 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.333028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.333085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.333159 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.334019 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.334696 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.335883 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.336287 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.336390 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.338372 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.338377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.338380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.338408 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.340606 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.341204 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.341208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.341210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.341339 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.341677 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.341682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.341685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.341957 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.342201 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.342525 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.342731 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:04:00.343647 17385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:00.427081 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17262
I20260620 07:04:00.440514 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17396
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:04:00.456462 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.441437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.11.81.194:45699)
0620 07:04:00.442104 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.442111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.442139 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:04:00.442198 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.442203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.442433 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.442567 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.443636 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.444705 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.446045 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.446608 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.446744 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.450670 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.450683 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.450697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.450734 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.453742 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.454476 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.454484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.454486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.454614 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.454959 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.454964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.454968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.455359 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.455595 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.455884 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.456166 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
I20260620 07:04:00.457743 17519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:00.539004 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17396
I20260620 07:04:00.553282 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17529
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:04:00.568100 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:00.554278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.11.81.195:45717)
0620 07:04:00.554884 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:00.554891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:00.554918 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:00.554974 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:00.554980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:00.555245 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:00.555422 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:00.556392 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.557375 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.558595 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:00.559153 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:00.559266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:00.563148 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:00.563160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:00.563176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:00.563213 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:00.565768 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.566366 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.566372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.566375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.566496 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:00.566783 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:00.566785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:00.566787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:00.567106 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:00.567362 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:00.567587 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 07:04:00.567783 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":308,"threads_started":1}
I20260620 07:04:00.569401 17652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:00.656039 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17529
I20260620 07:04:00.668351 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17169
I20260620 07:04:00.669611 17231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:00.771030 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17169
2026-06-20T07:04:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3787 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17684](info): set up 2 sockets
Jun 20 07:04:00 dist-test-slave-wvz3 krb5kdc[17684](info): commencing operation
krb5kdc: starting...
W20260620 07:04:01.775863 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:01 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab.
Entry for principal oryx/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab.
I20260620 07:04:01.982501 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39281
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab
--principal=oryx/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:02.310413 17697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:02.310704 17697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:02.310819 17697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:02.318169 17697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:02.318293 17697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:02.318351 17697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:02.318434 17697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:02.318483 17697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:02.328698 17697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab
--principal=oryx/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:02.330081 17697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:02.331440 17697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:02.343312 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:02.343220 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:02.343256 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:02.344221 17697 server_base.cc:1034] running on GCE node
I20260620 07:04:02.344980 17697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:02.346411 17697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:02.347611 17697 hybrid_clock.cc:648] HybridClock initialized: now 1781939042347556 us; error 45 us; skew 500 ppm
I20260620 07:04:02.347966 17697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:02 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939042, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:02.350955 17697 init.cc:377] Logged in from keytab as oryx/127.11.81.254@KRBTEST.COM (short username oryx)
I20260620 07:04:02.352576 17697 webserver.cc:460] Webserver started at http://127.11.81.254:44431/ using document root <none> and password file <none>
I20260620 07:04:02.353277 17697 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:02.353399 17697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:02.353693 17697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:02.356901 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "b8f6c007f42c44dfb4348a1ab0361b0d"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "d5e05aa3bd8517ec7c555470640f870a"
server_key_iv: "1c38cdedd4100550348f5af9ab867aaa"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.357784 17697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "b8f6c007f42c44dfb4348a1ab0361b0d"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "d5e05aa3bd8517ec7c555470640f870a"
server_key_iv: "1c38cdedd4100550348f5af9ab867aaa"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.363340 17697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:04:02.366994 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:02.368680 17697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:04:02.368870 17697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b8f6c007f42c44dfb4348a1ab0361b0d"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "d5e05aa3bd8517ec7c555470640f870a"
server_key_iv: "1c38cdedd4100550348f5af9ab867aaa"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.369078 17697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:02.387967 17697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:02.392774 17697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:02.393064 17697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:02.410871 17697 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:39281
I20260620 07:04:02.410950 17763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:39281 every 8 connection(s)
I20260620 07:04:02.412336 17697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:02.412928 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17697
I20260620 07:04:02.413107 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:02.413869 11591 external_mini_cluster.cc:1428] Setting key ffca708997af3dc6567f7e5a4e25ad20
I20260620 07:04:02.416642 17764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:04:02 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:02.429523 17764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Bootstrap starting.
I20260620 07:04:02.432898 17764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:02.434158 17764 log.cc:824] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:02.437552 17764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: No bootstrap required, opened a new log
I20260620 07:04:02.437852 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:02.415888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.11.81.254:39281)
0620 07:04:02.416653 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:02.416669 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:02.416722 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:04:02.418914 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:02.418941 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:02.419468 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:02.419935 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:02.421212 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.423250 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.424520 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.425525 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.425708 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:02.429749 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:02.429759 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:02.429774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:02.429827 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:02.433141 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:02.434047 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:02.434055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:02.434061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:02.434235 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:02.434726 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:02.434732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:02.434739 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:02.435583 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:02.435842 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:02.436407 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 07:04:02.436784 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260620 07:04:02.445698 17764 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } }
I20260620 07:04:02.446072 17764 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:02.446162 17764 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f6c007f42c44dfb4348a1ab0361b0d, State: Initialized, Role: FOLLOWER
I20260620 07:04:02.446689 17764 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } }
I20260620 07:04:02.446913 17764 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:02.447028 17764 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:02.447173 17764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:02.449581 17764 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } }
I20260620 07:04:02.450109 17764 leader_election.cc:302] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8f6c007f42c44dfb4348a1ab0361b0d; no voters:
I20260620 07:04:02.450767 17764 leader_election.cc:288] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:02.450894 17769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:02.451519 17769 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 LEADER]: Becoming Leader. State: Replica: b8f6c007f42c44dfb4348a1ab0361b0d, State: Running, Role: LEADER
I20260620 07:04:02.452069 17769 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } }
I20260620 07:04:02.452777 17764 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:02.454618 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f6c007f42c44dfb4348a1ab0361b0d. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } }
I20260620 07:04:02.455031 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:02.455847 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } }
I20260620 07:04:02.456254 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:02.458220 17778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:02.463078 17778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:02.471508 17778 catalog_manager.cc:1347] Generated new cluster ID: 2aa9a61faac244f686222bd4dfe7e779
I20260620 07:04:02.471635 17778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:02.480254 17778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:02.481832 17778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:02.493171 17778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Generated new TSK 0
I20260620 07:04:02.494047 17778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
I20260620 07:04:02.551260 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:02.874755 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:02.875073 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:02.875216 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:02.882658 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:02.882792 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:02.882954 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:02.892932 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:02.894225 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:02.895581 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:02.908360 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:02.908385 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:02.908442 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:02.909294 17790 server_base.cc:1034] running on GCE node
I20260620 07:04:02.909934 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:02.910991 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:02.912189 17790 hybrid_clock.cc:648] HybridClock initialized: now 1781939042912134 us; error 45 us; skew 500 ppm
I20260620 07:04:02.912519 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:02 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939042, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:02.915598 17790 init.cc:377] Logged in from keytab as oryx/127.11.81.193@KRBTEST.COM (short username oryx)
I20260620 07:04:02.917112 17790 webserver.cc:460] Webserver started at http://127.11.81.193:43675/ using document root <none> and password file <none>
I20260620 07:04:02.917809 17790 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:02.917927 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:02.918224 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:02.921135 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "1cc863e8db7d44ce81aedb7094c723e9"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "012e8e1c9fd1654e37925e30416527bb"
server_key_iv: "f019cf5480177d73f05fde287f524225"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.921911 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "1cc863e8db7d44ce81aedb7094c723e9"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "012e8e1c9fd1654e37925e30416527bb"
server_key_iv: "f019cf5480177d73f05fde287f524225"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.927054 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:04:02.930559 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:02.932044 17790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:02.932201 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "1cc863e8db7d44ce81aedb7094c723e9"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "012e8e1c9fd1654e37925e30416527bb"
server_key_iv: "f019cf5480177d73f05fde287f524225"
server_key_version: "encryptionkey@0"
I20260620 07:04:02.932386 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:02.946502 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:02.950577 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:02.950845 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:02.951853 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:02.953343 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:02.953420 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:02.953544 17790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:02.953600 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:02.983137 17790 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:41491
I20260620 07:04:02.983170 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:41491 every 8 connection(s)
I20260620 07:04:02.984462 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:02.993352 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17790
I20260620 07:04:02.993536 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:02.994144 11591 external_mini_cluster.cc:1428] Setting key 2b04a436b5fb4f641db8741a6b4f0d91
Jun 20 07:04:02 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939042, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:03.005447 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:02.986851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46355 (local address 127.11.81.254:39281)
0620 07:04:02.987263 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:02.987268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:02.988148 (+ 880us) server_negotiation.cc:407] Connection header received
0620 07:04:02.990139 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:02.990145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:02.990232 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:02.990335 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:02.991976 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.992811 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.994612 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.995030 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.995163 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:02.999090 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:02.999097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:02.999101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:02.999143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:03.002079 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:03.002977 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:03.002984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:03.002987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:03.003158 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:03.003710 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:03.003714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:03.003716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:03.004022 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:03.004150 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:03.005010 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 07:04:03.005187 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":215,"threads_started":1}
I20260620 07:04:03.006880 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:02.987314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:46355)
0620 07:04:02.987992 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:02.988036 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:02.989839 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:02.990585 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:02.990594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:02.991171 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:02.991805 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:02.991824 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.992980 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.992986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:02.994474 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:02.994487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:02.995254 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:02.995262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:02.995593 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:02.996309 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:02.996339 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:02.998843 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:03.002256 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:03.002266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:03.002282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:03.002805 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:03.003272 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:03.003276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:03.003302 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:03.003589 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:03.004145 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:03.004154 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:03.004606 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 07:04:03.004894 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 07:04:03.005450 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":254,"threads_started":1}
I20260620 07:04:03.008529 17918 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:03.008857 17918 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:03.009649 17918 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
I20260620 07:04:03.011960 17729 ts_manager.cc:194] Registered new tserver with Master: 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:03.013473 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:46355
WARNING: no policy specified for oryx/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
I20260620 07:04:03.042480 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:03.360811 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:03.361111 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:03.361235 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:03.368598 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:03.368718 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:03.368839 17924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:03.378569 17924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:03.379998 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:03.381352 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:03.394407 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:03.394434 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:03.394461 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:03.394738 17924 server_base.cc:1034] running on GCE node
I20260620 07:04:03.395597 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:03.396509 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:03.397744 17924 hybrid_clock.cc:648] HybridClock initialized: now 1781939043397672 us; error 53 us; skew 500 ppm
I20260620 07:04:03.398074 17924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:03 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939043, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:03.401273 17924 init.cc:377] Logged in from keytab as oryx/127.11.81.194@KRBTEST.COM (short username oryx)
I20260620 07:04:03.402783 17924 webserver.cc:460] Webserver started at http://127.11.81.194:35033/ using document root <none> and password file <none>
I20260620 07:04:03.403549 17924 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:03.403657 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:03.403911 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:03.406931 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "9b23d578b1e54bb0811dc3237b22f2b6"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "fed6852ddee581624c6e380baba3a4b0"
server_key_iv: "14e06b611b62bc10a2fc5315c3926c93"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.407743 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "9b23d578b1e54bb0811dc3237b22f2b6"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "fed6852ddee581624c6e380baba3a4b0"
server_key_iv: "14e06b611b62bc10a2fc5315c3926c93"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.413308 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:03.417088 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.418702 17924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:03.418867 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "9b23d578b1e54bb0811dc3237b22f2b6"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "fed6852ddee581624c6e380baba3a4b0"
server_key_iv: "14e06b611b62bc10a2fc5315c3926c93"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.419054 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:03.455935 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:03.460239 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:03.460510 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:03.461472 17924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:03.462793 17924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:03.462873 17924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.463002 17924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:03.463053 17924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.496946 17924 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:35789
I20260620 07:04:03.496992 18051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:35789 every 8 connection(s)
I20260620 07:04:03.498196 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:03.502632 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 17924
I20260620 07:04:03.502789 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:03.503183 11591 external_mini_cluster.cc:1428] Setting key d4fcaf07f4cfab486644122181898e9a
Jun 20 07:04:03 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939043, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:03.518155 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:03.500702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46623 (local address 127.11.81.254:39281)
0620 07:04:03.500856 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:03.500860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:03.501981 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 07:04:03.503958 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:03.503964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:03.504056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:03.504162 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:03.505964 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.506717 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.508410 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.508810 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.508918 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:03.512424 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:03.512429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:03.512433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:03.512483 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:03.515085 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:03.515872 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:03.515877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:03.515879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:03.516008 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:03.516420 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:03.516423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:03.516425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:03.516683 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:03.516812 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:03.517536 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 07:04:03.517811 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:04:03.519119 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:03.501135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:46623)
0620 07:04:03.501792 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:03.501834 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:03.503650 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:03.504417 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:03.504428 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:03.505190 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:03.505791 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:03.505808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.506873 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.506877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:03.508271 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:03.508282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.509025 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.509032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:03.509250 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:03.509846 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:03.509880 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:03.512164 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:03.515232 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:03.515242 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:03.515261 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:03.515739 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:03.516112 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:03.516118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:03.516121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:03.516308 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:03.516802 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:03.516811 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:03.517162 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 07:04:03.517415 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 07:04:03.517958 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260620 07:04:03.520689 18052 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:03.521036 18052 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:03.521970 18052 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:04:03.523732 17729 ts_manager.cc:194] Registered new tserver with Master: 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789)
I20260620 07:04:03.524688 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:46623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
I20260620 07:04:03.549897 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:03.866794 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:03.867086 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:03.867205 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:03.874470 18057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:03.874589 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:03.874737 18057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:03.884536 18057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:03.885841 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:03.887101 18057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:03.899811 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:03.899924 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:03.899951 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:03.900991 18057 server_base.cc:1034] running on GCE node
I20260620 07:04:03.901594 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:03.902532 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:03.903726 18057 hybrid_clock.cc:648] HybridClock initialized: now 1781939043903674 us; error 40 us; skew 500 ppm
I20260620 07:04:03.904053 18057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:03 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939043, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:03.907091 18057 init.cc:377] Logged in from keytab as oryx/127.11.81.195@KRBTEST.COM (short username oryx)
I20260620 07:04:03.908545 18057 webserver.cc:460] Webserver started at http://127.11.81.195:36903/ using document root <none> and password file <none>
I20260620 07:04:03.909219 18057 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:03.909315 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:03.909559 18057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:03.912483 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "b43b31c9d86049bb95e2bdc05dc28e7d"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "e685358af13f3f9b055912530b9b43ac"
server_key_iv: "7914bb8748f06a5a0ecf664781bf4523"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.913237 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "b43b31c9d86049bb95e2bdc05dc28e7d"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "e685358af13f3f9b055912530b9b43ac"
server_key_iv: "7914bb8748f06a5a0ecf664781bf4523"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.918618 18057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 07:04:03.922267 18072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.923990 18057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:03.924149 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b43b31c9d86049bb95e2bdc05dc28e7d"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "e685358af13f3f9b055912530b9b43ac"
server_key_iv: "7914bb8748f06a5a0ecf664781bf4523"
server_key_version: "encryptionkey@0"
I20260620 07:04:03.924329 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:03.947629 18057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:03.951783 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:03.952049 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:03.953044 18057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:03.954504 18057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:03.954578 18057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.954668 18057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:03.954711 18057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:03.984266 18057 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42437
I20260620 07:04:03.984277 18184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42437 every 8 connection(s)
I20260620 07:04:03.985806 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:03.989153 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18057
I20260620 07:04:03.989331 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:03.989852 11591 external_mini_cluster.cc:1428] Setting key ccaf1fa0db1515b12f73387921b16986
Jun 20 07:04:03 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939043, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:04.006348 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:03.988434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:58449 (local address 127.11.81.254:39281)
0620 07:04:03.988650 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:03.988656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:03.989858 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 07:04:03.992063 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:03.992067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:03.992132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:03.992208 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:03.993869 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.994691 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.996233 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.996649 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.996739 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:03.999943 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:03.999948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:03.999951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:03.999999 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.002666 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.003727 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.003732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.003736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.003952 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.004492 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.004495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.004498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.004795 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.004968 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.005926 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.006103 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:04:04.007490 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:03.988862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:58449)
0620 07:04:03.989681 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:03.989728 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:03.991751 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:03.992511 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:03.992523 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:03.993149 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:03.993718 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:03.993735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.994830 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.994835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:03.996122 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:03.996134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:03.996816 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:03.996820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:03.997023 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:03.997588 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:03.997609 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:03.999723 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.002828 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.002836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.002851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.003595 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.004076 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.004083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.004087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.004367 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.004955 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.004965 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.005459 (+ 494us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.005810 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.006276 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":298,"threads_started":1}
I20260620 07:04:04.008989 18185 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:04.009303 18185 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:04.010087 18185 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
I20260620 07:04:04.011579 17729 ts_manager.cc:194] Registered new tserver with Master: b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:04.012392 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:58449
I20260620 07:04:04.016762 17918 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
I20260620 07:04:04.018131 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.81.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.253.keytab.
Entry for principal oryx/127.11.81.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.253.keytab.
I20260620 07:04:04.063679 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:36195
--webserver_interface=127.11.81.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.253.keytab
--principal=oryx/127.11.81.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39281,127.11.81.253:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:04.383661 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:04.383992 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:04.384141 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:04.391507 18190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:04.391628 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:04.391700 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:04.391774 18190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:04.391832 18190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:04.401764 18190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39281,127.11.81.253:36195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.253.keytab
--principal=oryx/127.11.81.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.81.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
W20260620 07:04:04.402136 18190 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.81.254:39281,127.11.81.253:36195'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 07:04:04.403149 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:04.404548 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:04.416000 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:04.416023 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:04.416777 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:04.416939 18190 server_base.cc:1034] running on GCE node
I20260620 07:04:04.417734 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:04.419139 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:04.420365 18190 hybrid_clock.cc:648] HybridClock initialized: now 1781939044420294 us; error 42 us; skew 500 ppm
I20260620 07:04:04.420686 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939044, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:04.423795 18190 init.cc:377] Logged in from keytab as oryx/127.11.81.253@KRBTEST.COM (short username oryx)
I20260620 07:04:04.425303 18190 webserver.cc:460] Webserver started at http://127.11.81.253:46289/ using document root <none> and password file <none>
I20260620 07:04:04.425981 18190 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:04.426075 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:04.426362 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:04.429392 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/instance:
uuid: "f62d5048314542a0b09a31f3311198f4"
format_stamp: "Formatted at 2026-06-20 07:04:04 on dist-test-slave-wvz3"
server_key: "3ba24b9ba327ee763867355962317571"
server_key_iv: "b96e857754a8986ae919a0dbba4c1983"
server_key_version: "encryptionkey@0"
I20260620 07:04:04.430140 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal/instance:
uuid: "f62d5048314542a0b09a31f3311198f4"
format_stamp: "Formatted at 2026-06-20 07:04:04 on dist-test-slave-wvz3"
server_key: "3ba24b9ba327ee763867355962317571"
server_key_iv: "b96e857754a8986ae919a0dbba4c1983"
server_key_version: "encryptionkey@0"
I20260620 07:04:04.435261 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:04:04.438761 18205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:04.440152 18190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:04:04.440315 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "f62d5048314542a0b09a31f3311198f4"
format_stamp: "Formatted at 2026-06-20 07:04:04 on dist-test-slave-wvz3"
server_key: "3ba24b9ba327ee763867355962317571"
server_key_iv: "b96e857754a8986ae919a0dbba4c1983"
server_key_version: "encryptionkey@0"
I20260620 07:04:04.440517 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:04.460438 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:04.464689 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:04.464941 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:04.482409 18190 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.253:36195
I20260620 07:04:04.482478 18256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.253:36195 every 8 connection(s)
I20260620 07:04:04.483767 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
I20260620 07:04:04.487355 18257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:04.493002 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18190
I20260620 07:04:04.494000 18257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939044, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.253@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:04.514266 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.496787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.11.81.254:39281)
0620 07:04:04.496947 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.496953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.497846 (+ 893us) server_negotiation.cc:407] Connection header received
0620 07:04:04.498762 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.498767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.498850 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.498936 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.500458 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.501416 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.503230 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.503680 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.503809 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.507527 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.507535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.507540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.507583 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.510544 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.511575 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.511580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.511582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.511726 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.512205 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.512210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.512213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.512686 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.512812 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.513657 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.513925 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:04:04.515637 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.496844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47950)
0620 07:04:04.497696 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.497739 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.498487 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.499214 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.499220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.499898 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.500284 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.500311 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.501596 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.501601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.503077 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.503089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.503911 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.503922 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.504086 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.504801 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.504838 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.507274 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.510723 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.510735 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.510748 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.511433 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.511845 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.511850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.511853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.512076 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.512815 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.512824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.513240 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.513574 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.514117 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":1,"thread_start_us":427,"threads_started":1}
I20260620 07:04:04.515604 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.495388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.11.81.253:36195)
0620 07:04:04.496109 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.496119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.496155 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:04:04.498466 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.498486 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.498741 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.498961 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.500877 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.502216 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.503472 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.503988 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.504092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.508492 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.508505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.508541 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.508586 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.511845 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.512719 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.512727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.512730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.512878 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.513245 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.513250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.513252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.513630 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.513909 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.514238 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.514462 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":529,"thread_start_us":315,"threads_started":1}
I20260620 07:04:04.517733 18257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939044, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.253@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:04.527484 18052 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
I20260620 07:04:04.530412 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.518505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43798)
0620 07:04:04.518727 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.518746 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.518928 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.519214 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.519219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.519589 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.519734 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.519742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.520754 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.520758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.521623 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.521633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.522305 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.522310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.522470 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.523049 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.523066 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.525124 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.528043 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.528048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.528052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.528450 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.528804 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.528809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.528812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.528984 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.529523 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.529527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.529651 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.529769 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.530049 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:04:04.530489 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.518730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.11.81.253:36195)
0620 07:04:04.518926 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.518930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.518946 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:04:04.519007 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.519017 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.519071 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.519143 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.519871 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.520563 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.521735 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.522163 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.522251 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.525307 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.525312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.525316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.525344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.527913 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.528553 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.528559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.528563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.528710 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.529081 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.529085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.529088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.529418 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.529541 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.529863 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.530197 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":93}
I20260620 07:04:04.535494 18257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Bootstrap starting.
I20260620 07:04:04.538689 18257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:04.539740 18257 log.cc:824] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:04.542498 18257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: No bootstrap required, opened a new log
I20260620 07:04:04.549623 18257 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } }
I20260620 07:04:04.549963 18257 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:04.550048 18257 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62d5048314542a0b09a31f3311198f4, State: Initialized, Role: FOLLOWER
I20260620 07:04:04.550621 18257 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } }
I20260620 07:04:04.551477 18263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } }
I20260620 07:04:04.551766 18263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:04:04.552480 18257 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:04:04.561365 18274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:04:04.561576 18274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:04:04.563201 18190 master_runner.cc:428] Detected that this master f62d5048314542a0b09a31f3311198f4 is joining an existing cluster
I20260620 07:04:04.563339 18190 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.81.253:36195
I20260620 07:04:04.583652 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.567337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.11.81.254:39281)
0620 07:04:04.567510 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.567515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.568480 (+ 965us) server_negotiation.cc:407] Connection header received
0620 07:04:04.568754 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.568759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.568843 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.568914 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.570387 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.571546 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.573556 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.574219 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.574316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.576851 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.576858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.576864 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.576906 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.580389 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.581293 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.581299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.581303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.581462 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.581911 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.581916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.581920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.582273 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.582424 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.583010 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.583270 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:04:04.584794 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.567710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47958)
0620 07:04:04.568360 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.568383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.568596 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.569125 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.569130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.569782 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.570206 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.570221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.571732 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.571737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.573382 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.573398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.574601 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.574607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.574793 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.575422 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.575445 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.576632 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.580572 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.580577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.580582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.581136 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.581613 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.581620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.581624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.581807 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.582481 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.582486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.582687 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.583969 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.584321 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":364,"threads_started":1}
I20260620 07:04:04.595345 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.567722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43814)
0620 07:04:04.580005 (+ 12283us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.580022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.580171 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.580625 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.580629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.581410 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.581699 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.581708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.583046 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.583050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.584653 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.584754 (+ 101us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.585911 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.585917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.586062 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.586672 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.586694 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.587770 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.592502 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.592507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.592510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.593004 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.593401 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.593405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.593409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.593553 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.594230 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.594235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.594364 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.594490 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.594821 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12152,"mutex_wait_us":16,"spinlock_wait_cycles":20864,"thread_start_us":11947,"threads_started":1}
WARNING: no policy specified for oryx/127.11.81.252@KRBTEST.COM; defaulting to no policy
I20260620 07:04:04.596410 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.567453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.11.81.253:36195)
0620 07:04:04.567645 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.567651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.580137 (+ 12486us) server_negotiation.cc:407] Connection header received
0620 07:04:04.580288 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.580309 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.580387 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.580470 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.581829 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.582864 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.584887 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.585457 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.585527 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.587941 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.587947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.587951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.587986 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.592320 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.593129 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.593133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.593136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.593288 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.593648 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.593652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.593656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.594025 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.594185 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.595817 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.596054 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.252@KRBTEST.COM" created.
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939042, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.254@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:04.615383 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.593124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.11.81.253:36195)
0620 07:04:04.596926 (+ 3802us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.596932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.600278 (+ 3346us) server_negotiation.cc:407] Connection header received
0620 07:04:04.600651 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.600657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.600753 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.600853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.602193 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.603144 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.605113 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.605801 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.605924 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.609532 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.609537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.609540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.609574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.612540 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.613275 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.613279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.613281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.613414 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.613840 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.613844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.613846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.614120 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.614249 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.614741 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.614958 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3700,"thread_start_us":6328,"threads_started":1}
I20260620 07:04:04.615420 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.592771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43826)
0620 07:04:04.600127 (+ 7356us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.600188 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.600426 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.600921 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.600930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.601755 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.602025 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.602037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.603328 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.603333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.604951 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.604967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.605970 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.605976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.606206 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.606936 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.606959 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.609340 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.612686 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.612701 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.612710 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.613142 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.613557 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.613562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.613564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.613729 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.614266 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.614275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.614442 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.614604 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.614954 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7093,"thread_start_us":6938,"threads_started":1}
I20260620 07:04:04.617605 17729 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.81.253:36195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f62d5048314542a0b09a31f3311198f4"
member_type: NON_VOTER
last_known_addr {
host: "127.11.81.253"
port: 36195
}
attrs {
promote: true
}
}
dest_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d"
cas_config_opid_index: -1
I20260620 07:04:04.618566 17729 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } }
I20260620 07:04:04.622138 18288 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f62d5048314542a0b09a31f3311198f4 (127.11.81.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:04.623332 18232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:04.627663 18288 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.81.253:36195
I20260620 07:04:04.628029 18232 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 07:04:04.628839 18190 master.cc:537] Master@127.11.81.253:36195 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.252.keytab.
Entry for principal oryx/127.11.81.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.252.keytab.
I20260620 07:04:04.630923 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:38777
--webserver_interface=127.11.81.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.252.keytab
--principal=oryx/127.11.81.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:04:04.634363 18288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:04.634645 18288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:04.635078 18287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:04.643144 18190 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:04.643764 18190 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:04.644038 18190 tablet_replica.cc:330] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: stopping tablet replica
I20260620 07:04:04.651350 18190 master.cc:552] Master@127.11.81.253:36195 shutdown complete.
I20260620 07:04:04.651504 18190 master_runner.cc:305] Clearing existing system tablet
I20260620 07:04:04.651973 18190 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:04.652136 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 07:04:04.652715 17715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
I20260620 07:04:04.660816 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.660304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43838)
0620 07:04:04.660502 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.660634 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 07:04:04.661223 17715 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d -> Peer f62d5048314542a0b09a31f3311198f4 (127.11.81.253:36195): Couldn't send request to peer f62d5048314542a0b09a31f3311198f4. Status: Network error: Client connection negotiation failed: client connection to 127.11.81.253:36195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:04:04.661422 18190 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 07:04:04.663194 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:04.663599 18190 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:04.663892 18190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:04:04.664039 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "f62d5048314542a0b09a31f3311198f4"
format_stamp: "Formatted at 2026-06-20 07:04:04 on dist-test-slave-wvz3"
server_key: "3ba24b9ba327ee763867355962317571"
server_key_iv: "b96e857754a8986ae919a0dbba4c1983"
server_key_version: "encryptionkey@0"
I20260620 07:04:04.669230 18190 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:04.676440 18190 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:04:04.676643 18190 log.cc:1196] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 07:04:04.677071 18190 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Deleting consensus metadata
I20260620 07:04:04.677353 18190 master_runner.cc:315] Copying system tablet from 127.11.81.254:39281
I20260620 07:04:04.679869 18190 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: tablet copy: Beginning tablet copy session from remote peer at address 127.11.81.254:39281
I20260620 07:04:04.698235 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.680834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.11.81.254:39281)
0620 07:04:04.681026 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:04.681032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:04.683891 (+ 2859us) server_negotiation.cc:407] Connection header received
0620 07:04:04.684130 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:04.684135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:04.684231 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:04.684330 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:04.685411 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.686554 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.688428 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.689021 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.689163 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.691508 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:04.691515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:04.691520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:04.691563 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:04.694949 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.695849 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.695856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.695859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.696035 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:04.696485 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:04.696491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:04.696494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:04.696969 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:04.697137 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:04.697598 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 07:04:04.697860 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:04:04.698271 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:04.680629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47972)
0620 07:04:04.683732 (+ 3103us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:04.683754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:04.683941 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:04.684380 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:04.684385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:04.684919 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:04.685205 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.685228 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.686722 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.686728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.688254 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:04.688271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:04.689201 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:04.689215 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:04.689400 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:04.690114 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:04.690138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:04.691221 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:04.695129 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.695135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.695139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.695686 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.696159 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:04.696164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:04.696168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:04.696358 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:04.697095 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:04.697100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:04.697298 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 07:04:04.697443 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:04:04.697795 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":2804,"threads_started":1}
I20260620 07:04:04.699546 17749 tablet_copy_service.cc:140] P b8f6c007f42c44dfb4348a1ab0361b0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f62d5048314542a0b09a31f3311198f4 ({username='oryx', principal='oryx/127.11.81.253@KRBTEST.COM'} at 127.0.0.1:47972)
I20260620 07:04:04.699862 17749 tablet_copy_service.cc:161] P b8f6c007f42c44dfb4348a1ab0361b0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f62d5048314542a0b09a31f3311198f4 at {username='oryx', principal='oryx/127.11.81.253@KRBTEST.COM'} at 127.0.0.1:47972: session id = f62d5048314542a0b09a31f3311198f4-00000000000000000000000000000000
I20260620 07:04:04.704365 17749 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:04:04.706997 18190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:04:04.713953 18190 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: tablet copy: Starting download of 0 data blocks...
I20260620 07:04:04.714305 18190 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: tablet copy: Starting download of 1 WAL segments...
I20260620 07:04:04.717320 18190 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:04:04.721840 17749 tablet_copy_service.cc:342] P b8f6c007f42c44dfb4348a1ab0361b0d: Request end of tablet copy session f62d5048314542a0b09a31f3311198f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.81.253@KRBTEST.COM'} at 127.0.0.1:47972
I20260620 07:04:04.722084 17749 tablet_copy_service.cc:433] P b8f6c007f42c44dfb4348a1ab0361b0d: ending tablet copy session f62d5048314542a0b09a31f3311198f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f62d5048314542a0b09a31f3311198f4
I20260620 07:04:04.734004 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:04.736883 18301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:04.740623 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:04.751263 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:04.751823 18190 server_base.cc:1034] running on GCE node
I20260620 07:04:04.752245 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:04.752948 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:04.754189 18190 hybrid_clock.cc:648] HybridClock initialized: now 1781939044754108 us; error 34 us; skew 500 ppm
I20260620 07:04:04.754428 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:04 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939044, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:04.756573 18190 init.cc:377] Logged in from keytab as oryx/127.11.81.253@KRBTEST.COM (short username oryx)
I20260620 07:04:04.764165 18190 webserver.cc:460] Webserver started at http://127.11.81.253:32799/ using document root <none> and password file <none>
I20260620 07:04:04.764603 18190 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:04.764684 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:04.767834 18190 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 07:04:04.769766 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:04.770233 18190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 07:04:04.770373 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "f62d5048314542a0b09a31f3311198f4"
format_stamp: "Formatted at 2026-06-20 07:04:04 on dist-test-slave-wvz3"
server_key: "3ba24b9ba327ee763867355962317571"
server_key_iv: "b96e857754a8986ae919a0dbba4c1983"
server_key_version: "encryptionkey@0"
I20260620 07:04:04.770538 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:04.795682 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:04.799757 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:04.824184 18190 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.253:36195
I20260620 07:04:04.824241 18362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.253:36195 every 8 connection(s)
I20260620 07:04:04.825188 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
I20260620 07:04:04.830665 18363 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:04:04.834797 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Bootstrap starting.
I20260620 07:04:04.851943 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:04:04.852666 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: Bootstrap complete.
I20260620 07:04:04.853919 18363 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } }
I20260620 07:04:04.854212 18363 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f62d5048314542a0b09a31f3311198f4, State: Initialized, Role: LEARNER
I20260620 07:04:04.854476 18363 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } }
I20260620 07:04:04.855271 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:04.855597 18363 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:04.855712 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:04:04.866062 18377 catalog_manager.cc:1259] Loaded cluster ID: 2aa9a61faac244f686222bd4dfe7e779
I20260620 07:04:04.866204 18377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: loading cluster ID for follower catalog manager: success
I20260620 07:04:04.869995 18377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: acquiring CA information for follower catalog manager: success
I20260620 07:04:04.872771 18377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:04:05.015046 18185 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
W20260620 07:04:05.015866 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:05.016264 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:05.016414 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:05.024252 18291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:05.024370 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:05.024423 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:05.024478 18291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:05.024521 18291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:05.034626 18291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.252.keytab
--principal=oryx/127.11.81.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.81.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:05.036007 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:05.037281 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:05.048486 18383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:05.048491 18382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:05.048538 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:05.049510 18291 server_base.cc:1034] running on GCE node
I20260620 07:04:05.050311 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:05.051848 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:05.053053 18291 hybrid_clock.cc:648] HybridClock initialized: now 1781939045052992 us; error 55 us; skew 500 ppm
I20260620 07:04:05.053414 18291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:05.056504 18291 init.cc:377] Logged in from keytab as oryx/127.11.81.252@KRBTEST.COM (short username oryx)
I20260620 07:04:05.057982 18291 webserver.cc:460] Webserver started at http://127.11.81.252:41511/ using document root <none> and password file <none>
I20260620 07:04:05.058660 18291 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:05.058784 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:05.059080 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:05.062351 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/instance:
uuid: "4e7fbaddb09d4f92adcac937865aeccd"
format_stamp: "Formatted at 2026-06-20 07:04:05 on dist-test-slave-wvz3"
server_key: "18d667d51ba36ebea81108ad81d17d9f"
server_key_iv: "7b5266dcbc7b102072d1f1615fffe8e9"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.063148 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal/instance:
uuid: "4e7fbaddb09d4f92adcac937865aeccd"
format_stamp: "Formatted at 2026-06-20 07:04:05 on dist-test-slave-wvz3"
server_key: "18d667d51ba36ebea81108ad81d17d9f"
server_key_iv: "7b5266dcbc7b102072d1f1615fffe8e9"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.065969 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.060124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43846)
0620 07:04:05.060288 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.060307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.060406 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.061170 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.061175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.061206 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:05.061377 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.061386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.062639 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.062645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.064026 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.064041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.065160 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.065166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.065398 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.065451 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.065533 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.065680 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 07:04:05.066004 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.060334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.11.81.253:36195)
0620 07:04:05.060856 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.060861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.060880 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:04:05.060940 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.060944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.061009 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.061111 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:05.061501 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.062487 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.064171 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.064994 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.065108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.065339 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.065569 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.065700 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260620 07:04:05.067221 18338 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 07:04:05.067909 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f6c007f42c44dfb4348a1ab0361b0d. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:05.068174 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:04:05.068953 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:04:05.072597 18393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.074117 18291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:04:05.074282 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "4e7fbaddb09d4f92adcac937865aeccd"
format_stamp: "Formatted at 2026-06-20 07:04:05 on dist-test-slave-wvz3"
server_key: "18d667d51ba36ebea81108ad81d17d9f"
server_key_iv: "7b5266dcbc7b102072d1f1615fffe8e9"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.074472 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:05.091646 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:05.095822 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:05.096078 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:05.112829 18291 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.252:38777
I20260620 07:04:05.112890 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.252:38777 every 8 connection(s)
I20260620 07:04:05.114213 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
I20260620 07:04:05.117965 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:05.122303 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18291
I20260620 07:04:05.125473 18445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.252@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
I20260620 07:04:05.148504 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.129694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.11.81.254:39281)
0620 07:04:05.129889 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.129894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.130500 (+ 606us) server_negotiation.cc:407] Connection header received
0620 07:04:05.130703 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.130709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.130800 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.130900 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.132155 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.132905 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.134471 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.134995 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.135089 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.142312 (+ 7223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.142317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.142321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.142357 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.145143 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.146056 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.146059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.146062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.146187 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.146578 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.146583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.146587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.146966 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.147085 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.147946 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.148191 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:04:05.149605 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.124580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.11.81.252:38777)
0620 07:04:05.125390 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.125404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.125444 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:04:05.127469 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.127504 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.127943 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.128341 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.129983 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.131605 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.133025 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.133694 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.133821 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.143108 (+ 9287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.143121 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.143140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.143174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.146138 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.146897 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.146904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.146906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.147018 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.147437 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.147442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.147446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.147880 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.148158 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.148508 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.148731 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"spinlock_wait_cycles":20864,"thread_start_us":394,"threads_started":1}
I20260620 07:04:05.149667 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.129520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47974)
0620 07:04:05.130325 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.130383 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.130580 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.130965 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.130971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.131635 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.132010 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.132022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.133045 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.133049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.134336 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.134355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.135194 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.135219 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.135396 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.139242 (+ 3846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.139328 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.142092 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.145297 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.145307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.145321 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.145943 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.146300 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.146303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.146306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.146460 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.147080 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.147088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.147574 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.147825 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.148316 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"spinlock_wait_cycles":13184,"thread_start_us":372,"threads_started":1}
I20260620 07:04:05.151360 18445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.252@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:05.163720 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.152176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43862)
0620 07:04:05.152461 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.152479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.152590 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.152912 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.152915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.153216 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.153362 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.153370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.154328 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.154332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.155372 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.155383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.156090 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.156095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.156247 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.156807 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.156825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.158721 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.161550 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.161554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.161557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.161918 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.162193 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.162196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.162198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.162318 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.162974 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.162978 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.163100 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.163207 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.163477 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 07:04:05.163919 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.152159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.11.81.253:36195)
0620 07:04:05.152383 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.152389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.152527 (+ 138us) server_negotiation.cc:407] Connection header received
0620 07:04:05.152720 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.152726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.152798 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.152867 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.153474 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.154179 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.155497 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.155936 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.156060 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.158931 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.158937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.158940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.158974 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.161438 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.162001 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.162005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.162007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.162121 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.162389 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.162391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.162393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.162859 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.162983 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.163412 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.163600 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:04:05.164861 18445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.252@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
I20260620 07:04:05.176699 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.165608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.0.0.1:46252)
0620 07:04:05.165807 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.165823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.165912 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.166173 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.166177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.166511 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.166697 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.166706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.167667 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.167671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.168525 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.168536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.169172 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.169176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.169323 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.169814 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.169831 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.171782 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.174526 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.174530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.174533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.174885 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.175240 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.175244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.175246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.175455 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.175911 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.175914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.176030 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.176152 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.176374 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 07:04:05.176738 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.165555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.11.81.252:38777)
0620 07:04:05.165709 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.165713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.165875 (+ 162us) server_negotiation.cc:407] Connection header received
0620 07:04:05.165984 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.165987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.166038 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.166110 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.166821 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.167543 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.168632 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.169033 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.169131 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.171955 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.171960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.171963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.171990 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.174404 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.174989 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.174994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.174996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.175118 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.175552 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.175555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.175557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.175821 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.175922 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.176222 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.176478 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":53}
I20260620 07:04:05.181485 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Bootstrap starting.
I20260620 07:04:05.184571 18445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:05.185549 18445 log.cc:824] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:05.188117 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: No bootstrap required, opened a new log
I20260620 07:04:05.195097 18445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } }
I20260620 07:04:05.195479 18445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:05.195574 18445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7fbaddb09d4f92adcac937865aeccd, State: Initialized, Role: FOLLOWER
I20260620 07:04:05.196106 18445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } }
I20260620 07:04:05.196987 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } } }
I20260620 07:04:05.197332 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:04:05.198096 18445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:05.204658 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17790
W20260620 07:04:05.207787 18462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:04:05.208076 18462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:04:05.211272 18291 master_runner.cc:428] Detected that this master 4e7fbaddb09d4f92adcac937865aeccd is joining an existing cluster
I20260620 07:04:05.211428 18291 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.81.252:38777
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.193@KRBTEST.COM
I20260620 07:04:05.230893 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.206439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.11.81.193:41491)
0620 07:04:05.208675 (+ 2236us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.208684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.208725 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:04:05.208794 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.208802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.209069 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.209241 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.210886 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.212274 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.213674 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.214584 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.214701 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.221606 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.221628 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.221650 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.221726 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.226101 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.227133 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.227144 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.227148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.227466 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.227985 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.227991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.227995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.228574 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.228896 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.230200 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.230492 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065,"thread_start_us":1915,"threads_started":1}
I20260620 07:04:05.232748 17913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:05.249353 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.216878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.11.81.254:39281)
0620 07:04:05.217076 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.217083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.230432 (+ 13349us) server_negotiation.cc:407] Connection header received
0620 07:04:05.231153 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.231159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.231241 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.231341 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.232807 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.234019 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.236089 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.236724 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.236823 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.239444 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.239451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.239456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.239501 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.242880 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.244402 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.244408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.244411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.244589 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.245912 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.245918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.245921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.246315 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.246462 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.248728 (+ 2266us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.248992 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:04:05.251134 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.217118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.11.81.252:38777)
0620 07:04:05.217298 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.217303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.235146 (+ 17843us) server_negotiation.cc:407] Connection header received
0620 07:04:05.235700 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.235705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.235771 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.235837 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.237382 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.238461 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.240534 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.241204 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.241310 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.244799 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.244806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.244811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.244843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.248070 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.248773 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.248778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.248781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.248941 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.249352 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.249356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.249359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.249762 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.249893 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.250416 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.250766 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:04:05.252481 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.217519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.0.0.1:46258)
0620 07:04:05.235045 (+ 17526us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.235066 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.235579 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.235996 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.236000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.236830 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.237250 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.237262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.238786 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.238791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.240375 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.240410 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.241638 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.241644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.241816 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.242399 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.242424 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.244638 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.248183 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.248188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.248191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.248665 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.249058 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.249062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.249066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.249223 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.249955 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.249959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.250136 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.251598 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.251918 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17371,"thread_start_us":25777,"threads_started":1}
I20260620 07:04:05.252820 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.217257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47976)
0620 07:04:05.230313 (+ 13056us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.230335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.230976 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.231575 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.231579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.232200 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.232594 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.232610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.234226 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.234232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.235929 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.235941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.237125 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.237143 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.237315 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.238030 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.238052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.239167 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.243057 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.243061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.243065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.244253 (+ 1188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.244743 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.244747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.244751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.244954 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.246553 (+ 1599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.246558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.248408 (+ 1850us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.250707 (+ 2299us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.251016 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12854,"thread_start_us":17895,"threads_started":1}
I20260620 07:04:05.266124 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.220031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.11.81.253:36195)
0620 07:04:05.220231 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.220237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.243784 (+ 23547us) server_negotiation.cc:407] Connection header received
0620 07:04:05.245287 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.245294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.245385 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.245455 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.251261 (+ 5806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.252378 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.255701 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.256367 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.256475 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.258866 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.258873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.258877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.258918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.262859 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.263755 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.263761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.263765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.263929 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.264422 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.264427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.264430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.264792 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.264927 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.265511 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.265781 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:04:05.267474 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.219380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43872)
0620 07:04:05.243660 (+ 24280us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.243684 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.245108 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.245677 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.245681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.250138 (+ 4457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.250566 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.250582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.253129 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.253135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.255518 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.255538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.256795 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.256801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.256988 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.257650 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.257671 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.258624 (+ 953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.263016 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.263022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.263026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.263597 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.264076 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.264080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.264084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.264300 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.264998 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.265002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.265202 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.266667 (+ 1465us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.267002 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24103,"thread_start_us":11069,"threads_started":1}
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939042, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.254@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
I20260620 07:04:05.274683 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.259775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.0.0.1:46272)
0620 07:04:05.259996 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.260017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.260124 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.260653 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.260659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.261019 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.261174 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.261183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.262597 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.262605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.263973 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.263991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.265119 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.265125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.265305 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.265971 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.265995 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.268589 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.272129 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.272135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.272139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.272636 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.272990 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.272995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.272999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.273158 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.273729 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.273733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.273889 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.274010 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.274315 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 07:04:05.274755 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.260111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.11.81.252:38777)
0620 07:04:05.260277 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.260283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.260301 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:04:05.260394 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.260400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.260493 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.260602 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.261322 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.262430 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.264150 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.264825 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.264926 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.268834 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.268840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.268844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.268886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.271967 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.272755 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.272761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.272765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.272881 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.273265 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.273273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.273277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.273616 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.273742 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.274215 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.274451 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:04:05.276527 17729 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.81.252:38777: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd"
member_type: NON_VOTER
last_known_addr {
host: "127.11.81.252"
port: 38777
}
attrs {
promote: true
}
}
dest_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d"
cas_config_opid_index: 5
I20260620 07:04:05.277402 17729 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
I20260620 07:04:05.279247 18288 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.279724 18338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:04:05.281447 18420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:05.285247 18420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:04:05.292387 18451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:05.292701 18451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } }
I20260620 07:04:05.294291 18451 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281), f62d5048314542a0b09a31f3311198f4 (127.11.81.253:36195)
I20260620 07:04:05.295373 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e7fbaddb09d4f92adcac937865aeccd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" is_pre_election: true
I20260620 07:04:05.296680 18338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e7fbaddb09d4f92adcac937865aeccd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62d5048314542a0b09a31f3311198f4" is_pre_election: true
I20260620 07:04:05.298069 18396 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e7fbaddb09d4f92adcac937865aeccd; no voters: b8f6c007f42c44dfb4348a1ab0361b0d, f62d5048314542a0b09a31f3311198f4
I20260620 07:04:05.298858 18451 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:04:05.298696 18475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 07:04:05.302562 18474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:05.303050 18288 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.81.252:38777
I20260620 07:04:05.305192 18473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.305678 18473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:05.307420 18291 master.cc:537] Master@127.11.81.252:38777 shutting down...
I20260620 07:04:05.316864 18420 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
W20260620 07:04:05.317298 18420 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
I20260620 07:04:05.317276 18338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.321657 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.321993 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:04:05.322642 18473 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: attempting to promote NON_VOTER f62d5048314542a0b09a31f3311198f4 to VOTER
I20260620 07:04:05.323438 18473 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
I20260620 07:04:05.325600 18338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 LEARNER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 07:04:05.326195 18288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 07:04:05.328405 18473 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f62d5048314542a0b09a31f3311198f4 (127.11.81.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.329123 18338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f62d5048314542a0b09a31f3311198f4 (127.11.81.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.332669 18473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.332647 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.333079 18473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:05.333130 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:05.333655 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.333911 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:04:05.338188 17716 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d -> Peer 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252:38777): Couldn't send request to peer 4e7fbaddb09d4f92adcac937865aeccd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:04:05.347748 18420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f62d5048314542a0b09a31f3311198f4 (127.11.81.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:05.352396 18480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:05.352684 18480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:04:05.352687 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f6c007f42c44dfb4348a1ab0361b0d. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:05.352931 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:04:05.353219 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: NON_VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: true } } }
I20260620 07:04:05.353477 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:04:05.354265 18291 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 07:04:05.354658 18291 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 07:04:05.354815 18291 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: stopping tablet replica
I20260620 07:04:05.361788 18291 master.cc:552] Master@127.11.81.252:38777 shutdown complete.
I20260620 07:04:05.361871 18291 master_runner.cc:305] Clearing existing system tablet
I20260620 07:04:05.362191 18291 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:05.362294 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:05.364530 18291 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:04:05.365950 18486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.366291 18291 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:05.366552 18291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:04:05.366663 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "4e7fbaddb09d4f92adcac937865aeccd"
format_stamp: "Formatted at 2026-06-20 07:04:05 on dist-test-slave-wvz3"
server_key: "18d667d51ba36ebea81108ad81d17d9f"
server_key_iv: "7b5266dcbc7b102072d1f1615fffe8e9"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.369827 18291 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:05.375037 18291 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:04:05.375196 18291 log.cc:1196] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 07:04:05.375663 18291 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Deleting consensus metadata
I20260620 07:04:05.375885 18291 master_runner.cc:315] Copying system tablet from 127.11.81.254:39281
I20260620 07:04:05.380831 18291 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: tablet copy: Beginning tablet copy session from remote peer at address 127.11.81.254:39281
I20260620 07:04:05.394547 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.381580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.11.81.254:39281)
0620 07:04:05.381870 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.381876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.383834 (+ 1958us) server_negotiation.cc:407] Connection header received
0620 07:04:05.384010 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.384016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.384088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.384168 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.385037 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.385787 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.387057 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.387524 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.387640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.389384 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.389389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.389392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.389425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.391970 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.392615 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.392620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.392623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.392754 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.393114 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.393118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.393120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.393394 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.393527 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.393948 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.394161 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 07:04:05.394585 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.381508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:47980)
0620 07:04:05.383716 (+ 2208us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.383733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.383882 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.384232 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.384236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.384651 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.384879 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.384890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.385936 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.385940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.386912 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.386925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.387670 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.387675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.387867 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.388446 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.388464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.389193 (+ 729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.392107 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.392112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.392117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.392500 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.392862 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.392870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.392874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.393012 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.393506 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.393510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.393685 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.393811 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.394158 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"thread_start_us":1952,"threads_started":1}
I20260620 07:04:05.395519 17749 tablet_copy_service.cc:140] P b8f6c007f42c44dfb4348a1ab0361b0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4e7fbaddb09d4f92adcac937865aeccd ({username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:47980)
I20260620 07:04:05.395676 17749 tablet_copy_service.cc:161] P b8f6c007f42c44dfb4348a1ab0361b0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4e7fbaddb09d4f92adcac937865aeccd at {username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:47980: session id = 4e7fbaddb09d4f92adcac937865aeccd-00000000000000000000000000000000
I20260620 07:04:05.398041 17749 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:04:05.400113 18291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:04:05.404693 18291 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: tablet copy: Starting download of 0 data blocks...
I20260620 07:04:05.404970 18291 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: tablet copy: Starting download of 1 WAL segments...
I20260620 07:04:05.407061 18291 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:04:05.410140 17749 tablet_copy_service.cc:342] P b8f6c007f42c44dfb4348a1ab0361b0d: Request end of tablet copy session 4e7fbaddb09d4f92adcac937865aeccd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:47980
I20260620 07:04:05.410274 17749 tablet_copy_service.cc:433] P b8f6c007f42c44dfb4348a1ab0361b0d: ending tablet copy session 4e7fbaddb09d4f92adcac937865aeccd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4e7fbaddb09d4f92adcac937865aeccd
I20260620 07:04:05.421756 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:04:05.429559 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17790
W20260620 07:04:05.435096 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:05.445509 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:05.446998 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:41491
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:39281
--webserver_port=43675
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:05.447204 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:05.453027 18291 server_base.cc:1034] running on GCE node
I20260620 07:04:05.453397 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:05.454042 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:05.455361 18291 hybrid_clock.cc:648] HybridClock initialized: now 1781939045455310 us; error 36 us; skew 500 ppm
I20260620 07:04:05.455578 18291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:05.457499 18291 init.cc:377] Logged in from keytab as oryx/127.11.81.252@KRBTEST.COM (short username oryx)
I20260620 07:04:05.465015 18291 webserver.cc:460] Webserver started at http://127.11.81.252:37755/ using document root <none> and password file <none>
I20260620 07:04:05.465345 18291 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:05.465436 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:05.467937 18291 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:05.469563 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.469955 18291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 07:04:05.470078 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "4e7fbaddb09d4f92adcac937865aeccd"
format_stamp: "Formatted at 2026-06-20 07:04:05 on dist-test-slave-wvz3"
server_key: "18d667d51ba36ebea81108ad81d17d9f"
server_key_iv: "7b5266dcbc7b102072d1f1615fffe8e9"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.470254 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:05.488054 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:05.491509 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:05.519928 18291 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.252:38777
I20260620 07:04:05.519953 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.252:38777 every 8 connection(s)
I20260620 07:04:05.520879 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
I20260620 07:04:05.526485 18556 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:04:05.530385 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Bootstrap starting.
I20260620 07:04:05.550007 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:04:05.550462 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Bootstrap complete.
I20260620 07:04:05.551307 18556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
I20260620 07:04:05.551568 18556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e7fbaddb09d4f92adcac937865aeccd, State: Initialized, Role: LEARNER
I20260620 07:04:05.551823 18556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } }
I20260620 07:04:05.552852 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.553269 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: LEARNER
I20260620 07:04:05.553406 18556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:05.561688 18570 catalog_manager.cc:1259] Loaded cluster ID: 2aa9a61faac244f686222bd4dfe7e779
I20260620 07:04:05.561791 18570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: loading cluster ID for follower catalog manager: success
I20260620 07:04:05.564239 18570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: acquiring CA information for follower catalog manager: success
I20260620 07:04:05.565979 18570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:04:05.760183 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.753580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.11.81.252:38777)
0620 07:04:05.754143 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.754147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.754170 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:04:05.754228 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.754232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.754302 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.754398 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:05.754876 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.755976 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.758308 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.759155 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.759276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.759509 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.759804 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.759938 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
I20260620 07:04:05.760181 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.753369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.0.0.1:46278)
0620 07:04:05.753599 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.753622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.753789 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.754459 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.754464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.754493 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:05.754692 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.754704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.756120 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.756125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.758114 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.758133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.759360 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.759366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.759591 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.759627 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.759732 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.759892 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:04:05.761268 18531 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 07:04:05.761716 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f6c007f42c44dfb4348a1ab0361b0d. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:05.761972 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: LEARNER
W20260620 07:04:05.847270 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:05.847594 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:05.847739 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:05.855191 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:05.855350 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:05.855538 18499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:05.865883 18499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:41491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=43675
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:05.867242 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:05.868567 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:05.881764 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:05.881742 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:05.881705 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:05.882638 18499 server_base.cc:1034] running on GCE node
I20260620 07:04:05.883230 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:05.884191 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:05.885380 18499 hybrid_clock.cc:648] HybridClock initialized: now 1781939045885333 us; error 37 us; skew 500 ppm
I20260620 07:04:05.885679 18499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:05.888675 18499 init.cc:377] Logged in from keytab as oryx/127.11.81.193@KRBTEST.COM (short username oryx)
I20260620 07:04:05.890094 18499 webserver.cc:460] Webserver started at http://127.11.81.193:43675/ using document root <none> and password file <none>
I20260620 07:04:05.890764 18499 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:05.890887 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:05.896126 18499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:04:05.898871 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.900256 18499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:04:05.900399 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "1cc863e8db7d44ce81aedb7094c723e9"
format_stamp: "Formatted at 2026-06-20 07:04:02 on dist-test-slave-wvz3"
server_key: "012e8e1c9fd1654e37925e30416527bb"
server_key_iv: "f019cf5480177d73f05fde287f524225"
server_key_version: "encryptionkey@0"
I20260620 07:04:05.901458 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:05.915205 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:05.919410 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:05.919677 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:05.920621 18499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:05.921977 18499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:05.922060 18499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.922154 18499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:05.922220 18499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:05.951803 18499 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:41491
I20260620 07:04:05.951872 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:41491 every 8 connection(s)
I20260620 07:04:05.953315 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:05.963874 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18499
I20260620 07:04:05.964232 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17924
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.194@KRBTEST.COM
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
Jun 20 07:04:05 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939045, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
I20260620 07:04:05.999020 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.957081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.11.81.193:48109)
0620 07:04:05.973975 (+ 16894us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.974056 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.976877 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.980586 (+ 3709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.980592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.981331 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.982451 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.982479 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.983659 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.983664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.985368 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.985392 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.986555 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.986561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.986948 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.988257 (+ 1309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.988288 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.990952 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.994348 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.994359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.994378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.994857 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.995205 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.995210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.995214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.995481 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:05.996174 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:05.996182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:05.996654 (+ 472us) client_negotiation.cc:769] Sending connection context
0620 07:04:05.996954 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 07:04:05.997499 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16649,"mutex_wait_us":180,"spinlock_wait_cycles":56960,"thread_start_us":21855,"threads_started":1}
I20260620 07:04:05.999363 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.965624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.11.81.194:35789)
0620 07:04:05.977953 (+ 12329us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.977964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.977998 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:04:05.978105 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.978115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.978397 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.978576 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.980532 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.982117 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.983937 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.984767 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.984882 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.991908 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.991926 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.991943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.991987 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.996283 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.997201 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.997211 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.997214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.997376 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.997797 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.997803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.997806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.998143 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.998420 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.998723 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.998970 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12168,"thread_start_us":12012,"threads_started":1}
I20260620 07:04:06.000038 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.956563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:48109 (local address 127.11.81.253:36195)
0620 07:04:05.980210 (+ 23647us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.980215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.980242 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:05.980305 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.980310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.980392 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.980547 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.982670 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.983502 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.985521 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.986124 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.986255 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.991265 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.991272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.991277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.991343 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.994217 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.994972 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.994976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.994979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.995096 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.995609 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.995614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.995617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:05.995957 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:05.996092 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:05.999507 (+ 3415us) server_negotiation.cc:299] Negotiation successful
0620 07:04:05.999734 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23533,"thread_start_us":23413,"threads_started":1}
I20260620 07:04:06.000779 18047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:06.001884 18701 heartbeater.cc:347] Connected to a master server at 127.11.81.253:36195
I20260620 07:04:06.002200 18701 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.002384 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.956251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:33701 (local address 127.11.81.252:38777)
0620 07:04:05.956914 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.956922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.979377 (+ 22455us) server_negotiation.cc:407] Connection header received
0620 07:04:05.979606 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.979611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.979681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.979740 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.982654 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.983749 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.989134 (+ 5385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.989781 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.989881 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.995474 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.995480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.995485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.995524 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.998584 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:05.999948 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:05.999953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:05.999957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.000232 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.000667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.000672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.000676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.001028 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.001176 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.001750 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.002011 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483}
I20260620 07:04:06.003068 18701 heartbeater.cc:511] Master 127.11.81.253:36195 requested a full tablet report, sending...
I20260620 07:04:06.003484 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.957425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.11.81.193:33701)
0620 07:04:05.979252 (+ 21827us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.979270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.979475 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.980816 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.980827 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.981433 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.982451 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.982479 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.987364 (+ 4885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.987369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.988905 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.988920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.990262 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.990269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.990464 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.991102 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.991125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.994511 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.999377 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.999382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.999386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.999807 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.000337 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.000342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.000345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.000515 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.001261 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:06.001267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:06.001429 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 07:04:06.002578 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0620 07:04:06.002930 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21709,"mutex_wait_us":72,"spinlock_wait_cycles":3968,"thread_start_us":17882,"threads_started":1}
I20260620 07:04:06.004724 18699 heartbeater.cc:347] Connected to a master server at 127.11.81.252:38777
I20260620 07:04:06.004865 18699 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.005324 18699 heartbeater.cc:511] Master 127.11.81.252:38777 requested a full tablet report, sending...
I20260620 07:04:06.005818 18328 ts_manager.cc:194] Registered new tserver with Master: 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:06.007169 18521 ts_manager.cc:194] Registered new tserver with Master: 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:06.009258 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.956924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:42279)
0620 07:04:05.977333 (+ 20409us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:05.977352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:05.977452 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:05.980216 (+ 2764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:05.980228 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:05.981373 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:05.982760 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.982771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.985528 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.985532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.986987 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:05.987000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.988018 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.988025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:05.988183 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.988775 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:05.988797 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:05.991691 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:05.998755 (+ 7064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:05.998760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:05.998764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:05.999251 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.007403 (+ 8152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.007409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.007413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.007598 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.008227 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:06.008232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:06.008374 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 07:04:06.008496 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:04:06.008813 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8176,"mutex_wait_us":140,"spinlock_wait_cycles":19328,"thread_start_us":7928,"threads_started":1}
I20260620 07:04:06.009261 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:05.957142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:42279 (local address 127.11.81.254:39281)
0620 07:04:05.979510 (+ 22368us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:05.979516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:05.979532 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:04:05.979586 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:05.979590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:05.979658 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:05.979748 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:05.983676 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.984740 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.987356 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:05.987833 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:05.987946 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:05.992075 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:05.992083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:05.992088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:05.992130 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:05.997437 (+ 5307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.005166 (+ 7729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.005172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.005174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.005332 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.007748 (+ 2416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.007754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.007756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.008095 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.008257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.008638 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.008885 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22249,"thread_start_us":22152,"threads_started":1}
I20260620 07:04:06.010356 18700 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:06.010528 18700 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.010964 18700 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
I20260620 07:04:06.012054 17729 ts_manager.cc:194] Re-registered known tserver with Master: 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:06.012869 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:42279
I20260620 07:04:06.086596 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17924
I20260620 07:04:06.100036 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:35789
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:39281
--webserver_port=35033
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:04:06.142421 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: NON_VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: true } } }
I20260620 07:04:06.142831 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:06.144407 18712 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: attempting to promote NON_VOTER 4e7fbaddb09d4f92adcac937865aeccd to VOTER
I20260620 07:04:06.144975 18712 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } }
I20260620 07:04:06.146898 18338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 07:04:06.146936 18531 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 LEARNER]: Refusing update from remote peer b8f6c007f42c44dfb4348a1ab0361b0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 07:04:06.147550 18713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 07:04:06.147960 18712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 07:04:06.150415 18712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.151078 18338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.155055 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.155381 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:06.155570 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.155771 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:06.159646 18716 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.160153 18716 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:04:06.159989 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.160255 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:06.161083 18531 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.164952 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:06.165211 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:04:06.430907 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:06.431213 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:06.431380 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:06.438834 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:06.438969 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:06.439124 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:06.449231 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:35789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=35033
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:06.450670 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:06.452159 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:06.465298 18722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:06.465324 18725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:06.465549 18723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:06.466333 18711 server_base.cc:1034] running on GCE node
I20260620 07:04:06.466923 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:06.467887 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:06.469137 18711 hybrid_clock.cc:648] HybridClock initialized: now 1781939046469068 us; error 58 us; skew 500 ppm
I20260620 07:04:06.469440 18711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:06 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939046, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:06.472821 18711 init.cc:377] Logged in from keytab as oryx/127.11.81.194@KRBTEST.COM (short username oryx)
I20260620 07:04:06.474328 18711 webserver.cc:460] Webserver started at http://127.11.81.194:35033/ using document root <none> and password file <none>
I20260620 07:04:06.475010 18711 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:06.475122 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:06.480603 18711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 07:04:06.483451 18732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:06.484825 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:06.484978 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "9b23d578b1e54bb0811dc3237b22f2b6"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "fed6852ddee581624c6e380baba3a4b0"
server_key_iv: "14e06b611b62bc10a2fc5315c3926c93"
server_key_version: "encryptionkey@0"
I20260620 07:04:06.486011 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:06.514456 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:06.518987 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:06.519275 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:06.520330 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:06.521809 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:06.521904 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:06.522010 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:06.522076 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:06.554463 18711 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:35789
I20260620 07:04:06.554538 18844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:35789 every 8 connection(s)
I20260620 07:04:06.556027 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:06.560469 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18711
I20260620 07:04:06.560763 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18057
Jun 20 07:04:06 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.195@KRBTEST.COM
Jun 20 07:04:06 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939046, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
Jun 20 07:04:06 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939046, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
Jun 20 07:04:06 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939046, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:06.600224 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.558922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46999 (local address 127.11.81.253:36195)
0620 07:04:06.571623 (+ 12701us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:06.571627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:06.580029 (+ 8402us) server_negotiation.cc:407] Connection header received
0620 07:04:06.580150 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:06.580156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:06.580241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:06.580337 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:06.583045 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584113 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587135 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.587779 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587882 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.593697 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:06.593703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:06.593708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:06.593768 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:06.596357 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.597751 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.597756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.597760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.597915 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.598348 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.598353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.598356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.598696 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.598836 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.599609 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.599859 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12596,"thread_start_us":12500,"threads_started":1}
I20260620 07:04:06.600443 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.558922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:58281 (local address 127.11.81.252:38777)
0620 07:04:06.570003 (+ 11081us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:06.570010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:06.579486 (+ 9476us) server_negotiation.cc:407] Connection header received
0620 07:04:06.580311 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:06.580316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:06.580389 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:06.580458 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:06.583025 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584210 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587107 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.587789 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587886 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.593053 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:06.593059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:06.593063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:06.593112 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:06.596259 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.597481 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.597486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.597489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.597639 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.598153 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.598159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.598161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.598477 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.598612 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.599872 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.600101 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10892,"thread_start_us":11106,"threads_started":1}
I20260620 07:04:06.601325 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.558920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:35037 (local address 127.11.81.254:39281)
0620 07:04:06.574506 (+ 15586us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:06.574513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:06.577164 (+ 2651us) server_negotiation.cc:407] Connection header received
0620 07:04:06.579963 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:06.579968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:06.580065 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:06.580161 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:06.583035 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584210 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587107 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.587784 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.587886 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.592667 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:06.592673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:06.592676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:06.592717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:06.596184 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.597397 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.597404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.597408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.597570 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.598938 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.598943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.598947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.599315 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.599454 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.600887 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.601072 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15447,"thread_start_us":15591,"threads_started":1}
I20260620 07:04:06.601593 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.562867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.11.81.194:46999)
0620 07:04:06.579909 (+ 17042us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:06.579925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:06.580033 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:06.580592 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:06.580604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:06.581763 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:06.582823 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.582851 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584337 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.584342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.586955 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.586972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.588285 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.588294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.588692 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.589665 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:06.589687 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:06.593502 (+ 3815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:06.596471 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.596475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.596484 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.597284 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.598073 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.598077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.598080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.598231 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.598911 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:06.598915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:06.599182 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 07:04:06.599510 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 07:04:06.600069 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16943,"mutex_wait_us":570,"spinlock_wait_cycles":2304,"thread_start_us":16191,"threads_started":1}
I20260620 07:04:06.602196 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.562461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.11.81.194:58281)
0620 07:04:06.579375 (+ 16914us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:06.579400 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:06.580177 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:06.580664 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:06.580668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:06.581763 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:06.582839 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.582851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584371 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.584377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.586957 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.586967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.588255 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.588263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.588677 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.589522 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:06.589544 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:06.592844 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:06.596393 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.596402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.596420 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.597328 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.597759 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.597764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.597768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.598045 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.598618 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:06.598626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:06.599591 (+ 965us) client_negotiation.cc:769] Sending connection context
0620 07:04:06.601502 (+ 1911us) client_negotiation.cc:240] Negotiation successful
0620 07:04:06.601777 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16758,"mutex_wait_us":483,"spinlock_wait_cycles":62848,"thread_start_us":14003,"threads_started":1}
I20260620 07:04:06.602988 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.562101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.11.81.195:42437)
0620 07:04:06.575869 (+ 13768us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:06.575878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:06.576235 (+ 357us) server_negotiation.cc:407] Connection header received
0620 07:04:06.576372 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:06.576382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:06.576749 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:06.576941 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:06.578944 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.580451 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.582248 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.583092 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.583207 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.590124 (+ 6917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:06.590143 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:06.590162 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:06.590208 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:06.597905 (+ 7697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.598798 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.598809 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.598812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.598970 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:06.599406 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:06.599411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:06.599413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:06.599815 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:06.600106 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:06.602366 (+ 2260us) server_negotiation.cc:299] Negotiation successful
0620 07:04:06.602608 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13597,"thread_start_us":13414,"threads_started":1}
I20260620 07:04:06.603385 18845 heartbeater.cc:347] Connected to a master server at 127.11.81.252:38777
I20260620 07:04:06.603427 18847 heartbeater.cc:347] Connected to a master server at 127.11.81.253:36195
I20260620 07:04:06.603613 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:06.559596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:35037)
0620 07:04:06.576857 (+ 17261us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:06.576923 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:06.579487 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:06.580590 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:06.580603 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:06.581763 (+ 1160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:06.582828 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.582851 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.584371 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.584377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.586925 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:06.586942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:06.588255 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:06.588263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:06.588707 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:06.589479 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:06.589513 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:06.592438 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:06.596394 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.596403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.596419 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.597284 (+ 865us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.597769 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:06.597773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:06.597776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:06.598842 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:06.600179 (+ 1337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:06.600183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:06.600332 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 07:04:06.600466 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 07:04:06.600769 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16976,"mutex_wait_us":401,"spinlock_wait_cycles":43008,"thread_start_us":19915,"threads_started":1}
I20260620 07:04:06.603734 18847 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.603739 18845 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.604485 18846 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:06.604628 18846 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:06.604751 18847 heartbeater.cc:511] Master 127.11.81.253:36195 requested a full tablet report, sending...
I20260620 07:04:06.605094 18846 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
I20260620 07:04:06.605561 18845 heartbeater.cc:511] Master 127.11.81.252:38777 requested a full tablet report, sending...
I20260620 07:04:06.606166 18180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:06.606274 17729 ts_manager.cc:194] Re-registered known tserver with Master: 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789)
I20260620 07:04:06.606691 18521 ts_manager.cc:194] Registered new tserver with Master: 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789)
I20260620 07:04:06.606853 18328 ts_manager.cc:194] Registered new tserver with Master: 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789)
I20260620 07:04:06.607497 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:35037
I20260620 07:04:06.692499 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18057
I20260620 07:04:06.706699 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:42437
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:39281
--webserver_port=36903
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:04:07.015393 18700 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
W20260620 07:04:07.031558 18858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:07.031862 18858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:07.031998 18858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:07.039207 18858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:07.039364 18858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:07.039515 18858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:07.049422 18858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:42437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=36903
--tserver_master_addrs=127.11.81.254:39281,127.11.81.253:36195,127.11.81.252:38777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:07.050827 18858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:07.052305 18858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:07.068490 18863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:07.068502 18864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:07.077847 18858 server_base.cc:1034] running on GCE node
W20260620 07:04:07.078948 18866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:07.079958 18858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:07.081331 18858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:07.082612 18858 hybrid_clock.cc:648] HybridClock initialized: now 1781939047082492 us; error 100 us; skew 500 ppm
I20260620 07:04:07.083048 18858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:07 dist-test-slave-wvz3 krb5kdc[17684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939047, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:07.087421 18858 init.cc:377] Logged in from keytab as oryx/127.11.81.195@KRBTEST.COM (short username oryx)
I20260620 07:04:07.089388 18858 webserver.cc:460] Webserver started at http://127.11.81.195:36903/ using document root <none> and password file <none>
I20260620 07:04:07.090381 18858 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:07.090536 18858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:07.098152 18858 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:07.102181 18873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:07.104295 18858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:04:07.104496 18858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b43b31c9d86049bb95e2bdc05dc28e7d"
format_stamp: "Formatted at 2026-06-20 07:04:03 on dist-test-slave-wvz3"
server_key: "e685358af13f3f9b055912530b9b43ac"
server_key_iv: "7914bb8748f06a5a0ecf664781bf4523"
server_key_version: "encryptionkey@0"
I20260620 07:04:07.106045 18858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:07.160091 18858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:07.165974 18858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:07.166337 18858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:07.167735 18858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:07.169656 18858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:07.169778 18858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:07.169885 18858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:07.169960 18858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:07.208701 18858 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42437
I20260620 07:04:07.208722 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42437 every 8 connection(s)
I20260620 07:04:07.210645 18858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:07.217281 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 18858
Jun 20 07:04:07 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939047, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.252@KRBTEST.COM
Jun 20 07:04:07 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939047, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:07.288811 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.215850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.11.81.195:45533)
0620 07:04:07.249697 (+ 33847us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.249726 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.249890 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.252302 (+ 2412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.252316 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.253299 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:07.254332 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.254360 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.256320 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.256326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.258754 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.258774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.260241 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.260249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.260569 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.262192 (+ 1623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:07.262229 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:07.268630 (+ 6401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:07.280188 (+ 11558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.280194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.280198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.281349 (+ 1151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.283067 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.283073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.283077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.283389 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.285852 (+ 2463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:07.285861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:07.286339 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.286666 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.287232 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33691,"thread_start_us":34718,"threads_started":1}
I20260620 07:04:07.289156 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.213992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:45533 (local address 127.11.81.252:38777)
0620 07:04:07.251488 (+ 37496us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.251495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.251517 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:04:07.251583 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.251599 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.251682 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.251807 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.254569 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.255801 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.258952 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.259735 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.259846 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.268918 (+ 9072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.268926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.268932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.268981 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.272773 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.281505 (+ 8732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.281511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.281515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.281688 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.283546 (+ 1858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.283551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.283556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.283925 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.284066 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.288293 (+ 4227us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.288528 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37363,"thread_start_us":37197,"threads_started":1}
I20260620 07:04:07.290783 18986 heartbeater.cc:347] Connected to a master server at 127.11.81.252:38777
I20260620 07:04:07.291163 18986 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:07.292157 18986 heartbeater.cc:511] Master 127.11.81.252:38777 requested a full tablet report, sending...
I20260620 07:04:07.292447 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.231803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:45801)
0620 07:04:07.253641 (+ 21838us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.253663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.253794 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.254495 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.254499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.255129 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:07.255513 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.255526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.257228 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.257234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.260376 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.260392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.264077 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.264084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.264281 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.265289 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:07.265313 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:07.269460 (+ 4147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:07.279253 (+ 9793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.279266 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.279307 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.280619 (+ 1312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.284924 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.284930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.284934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.285099 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.291277 (+ 6178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:07.291308 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:07.291475 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.291669 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.291969 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21669,"thread_start_us":18406,"threads_started":1}
Jun 20 07:04:07 dist-test-slave-wvz3 krb5kdc[17684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939047, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.253@KRBTEST.COM
I20260620 07:04:07.294173 18521 ts_manager.cc:194] Registered new tserver with Master: b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:07.294425 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.231638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:45801 (local address 127.11.81.254:39281)
0620 07:04:07.250550 (+ 18912us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.250557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.253911 (+ 3354us) server_negotiation.cc:407] Connection header received
0620 07:04:07.253993 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.253998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.254111 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.254302 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.255758 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.257043 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.261090 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.261736 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.261862 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.273655 (+ 11793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.273661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.273667 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.273714 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.279062 (+ 5348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.284249 (+ 5187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.284256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.284260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.284442 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.285275 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.285281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.285285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.285670 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.291407 (+ 5737us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.291790 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.292035 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18785,"thread_start_us":18672,"threads_started":1}
I20260620 07:04:07.295476 18987 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39281
I20260620 07:04:07.295674 18987 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:07.296195 18987 heartbeater.cc:511] Master 127.11.81.254:39281 requested a full tablet report, sending...
I20260620 07:04:07.297196 17729 ts_manager.cc:194] Re-registered known tserver with Master: b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:07.298281 17729 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:45801
I20260620 07:04:07.303454 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.215439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:34557 (local address 127.11.81.253:36195)
0620 07:04:07.252539 (+ 37100us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.252545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.252569 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:04:07.252635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.252642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.252724 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.252841 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.260620 (+ 7779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.261837 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.265554 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.266258 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.266363 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.296325 (+ 29962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.296332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.296338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.296381 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.300186 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.301188 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.301194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.301197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.301365 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.301829 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.301834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.301838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.302168 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.302299 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.302845 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.303066 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36958,"thread_start_us":37526,"threads_started":1}
I20260620 07:04:07.304428 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.214740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.11.81.195:34557)
0620 07:04:07.246924 (+ 32184us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.247003 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.249741 (+ 2738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.258152 (+ 8411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.258157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.258688 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:07.258961 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.258972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.262051 (+ 3079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.262061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.265342 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.265359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.282342 (+ 16983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.282349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.282576 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.289966 (+ 7390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:07.289991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:07.296059 (+ 6068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:07.300341 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.300347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.300351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.300813 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.301514 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:07.301519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:07.301522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:07.301699 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:07.302380 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:07.302385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:07.302542 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.303870 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.304163 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31738,"thread_start_us":42910,"threads_started":1}
I20260620 07:04:07.305325 18988 heartbeater.cc:347] Connected to a master server at 127.11.81.253:36195
I20260620 07:04:07.305465 18988 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:07.305907 18988 heartbeater.cc:511] Master 127.11.81.253:36195 requested a full tablet report, sending...
I20260620 07:04:07.307169 18328 ts_manager.cc:194] Registered new tserver with Master: b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:07.312728 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.266827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.11.81.254:39281)
0620 07:04:07.294657 (+ 27830us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.294662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.294676 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:04:07.294717 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.294721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.294789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.294886 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.296777 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.297816 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.299581 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.300164 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.300276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.304195 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.304202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.304207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.304253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.308080 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.309574 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.309580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.309583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.309732 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.310199 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.310204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.310208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.310580 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.310833 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.312074 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.312361 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27724,"thread_start_us":35501,"threads_started":1}
I20260620 07:04:07.315837 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.270292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.11.81.252:38777)
0620 07:04:07.279726 (+ 9434us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.279733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.284792 (+ 5059us) server_negotiation.cc:407] Connection header received
0620 07:04:07.285292 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.285299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.285393 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.285470 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.287969 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.289186 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.301992 (+ 12806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.302625 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.302725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.308053 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.308061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.308066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.308109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.311776 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.312738 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.312745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.312749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.312907 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.313365 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.313370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.313374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.313731 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.313955 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.314547 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.315088 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9278,"thread_start_us":9129,"threads_started":1}
I20260620 07:04:07.319175 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.279946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.11.81.253:36195)
0620 07:04:07.288266 (+ 8320us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.288272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.289844 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 07:04:07.289914 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.289920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.289998 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.290102 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.297262 (+ 7160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.300283 (+ 3021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.307801 (+ 7518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.308419 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.308527 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.311487 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.311494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.311498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.311538 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.315723 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.316884 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.316890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.316894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.317073 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.317528 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.317534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.317537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.317878 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.318131 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.318526 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.318774 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8200,"thread_start_us":8031,"threads_started":1}
I20260620 07:04:07.317291 17729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:07.325737 17729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:07.379621 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.350164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41491 (local address 127.0.0.1:47308)
0620 07:04:07.360256 (+ 10092us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.360277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.360416 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.372455 (+ 12039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.372462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.372493 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:07.372902 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.372915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.375080 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.375085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.377048 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.377060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.378727 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.378734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.378885 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.378918 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.379014 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.379162 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9940,"mutex_wait_us":133,"thread_start_us":9443,"threads_started":1}
I20260620 07:04:07.380168 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.350916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.11.81.194:35789)
0620 07:04:07.371988 (+ 21072us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.371997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.372025 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:04:07.372147 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.372155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.372414 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.372616 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:07.373651 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.375058 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.377452 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.378686 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.378800 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.378961 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.379710 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.379888 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20904,"thread_start_us":21929,"threads_started":1}
I20260620 07:04:07.380580 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.350697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35789 (local address 127.0.0.1:47118)
0620 07:04:07.371396 (+ 20699us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.371418 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.371567 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.373060 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.373064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.373089 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:07.373460 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.373472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.375182 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.375191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.377305 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.377319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.379040 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.379045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.379199 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.379231 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.379385 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.379514 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20541,"mutex_wait_us":53,"thread_start_us":20396,"threads_started":1}
I20260620 07:04:07.380805 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.351145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42437 (local address 127.0.0.1:39738)
0620 07:04:07.373252 (+ 22107us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.373272 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.373455 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.374006 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.374011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.374032 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:07.374368 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.374378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.376183 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.376188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.377926 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.377937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.379925 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.379930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.380119 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.380151 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.380232 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.380354 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21941,"mutex_wait_us":3,"thread_start_us":21777,"threads_started":1}
I20260620 07:04:07.381841 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.350635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.11.81.193:41491)
0620 07:04:07.370794 (+ 20159us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.370804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.370841 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:04:07.371777 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.371788 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.372180 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.373172 (+ 992us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:07.373406 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.374943 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.377194 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.378550 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.378678 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.378910 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.380729 (+ 1819us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.381627 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19975,"thread_start_us":20725,"threads_started":1}
I20260620 07:04:07.382319 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.351554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.11.81.195:42437)
0620 07:04:07.368839 (+ 17285us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.368850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.373402 (+ 4552us) server_negotiation.cc:407] Connection header received
0620 07:04:07.373625 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.373634 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.373846 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.374005 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:07.374547 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.376022 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.378066 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.379521 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.379651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.380139 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.380538 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.382097 (+ 1559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17070,"thread_start_us":16854,"threads_started":1}
I20260620 07:04:07.384471 18780 tablet_service.cc:1467] Processing CreateTablet for tablet b01876c8e3004e7990857b4cd1d6e97d (DEFAULT_TABLE table=test-table [id=8ec8fb27f71746e19094ea6421f0d470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:07.384830 18634 tablet_service.cc:1467] Processing CreateTablet for tablet b01876c8e3004e7990857b4cd1d6e97d (DEFAULT_TABLE table=test-table [id=8ec8fb27f71746e19094ea6421f0d470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:07.385895 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01876c8e3004e7990857b4cd1d6e97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:07.385919 18780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01876c8e3004e7990857b4cd1d6e97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:07.385576 18921 tablet_service.cc:1467] Processing CreateTablet for tablet b01876c8e3004e7990857b4cd1d6e97d (DEFAULT_TABLE table=test-table [id=8ec8fb27f71746e19094ea6421f0d470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:07.386818 18921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01876c8e3004e7990857b4cd1d6e97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:07.397312 19023 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Bootstrap starting.
I20260620 07:04:07.400786 19023 tablet_bootstrap.cc:654] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:07.401789 19023 log.cc:824] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:07.403409 19025 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Bootstrap starting.
I20260620 07:04:07.404445 19024 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Bootstrap starting.
I20260620 07:04:07.405019 19023 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: No bootstrap required, opened a new log
I20260620 07:04:07.405314 19023 ts_tablet_manager.cc:1397] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:04:07.408229 19024 tablet_bootstrap.cc:654] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:07.409448 19024 log.cc:824] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:07.411082 19025 tablet_bootstrap.cc:654] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:07.412438 19025 log.cc:824] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:07.413398 19024 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: No bootstrap required, opened a new log
I20260620 07:04:07.413738 19024 ts_tablet_manager.cc:1397] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 07:04:07.415501 19023 raft_consensus.cc:348] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.415936 19023 raft_consensus.cc:374] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:07.416030 19023 raft_consensus.cc:729] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc863e8db7d44ce81aedb7094c723e9, State: Initialized, Role: FOLLOWER
I20260620 07:04:07.416728 19023 consensus_queue.cc:260] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.425480 19024 raft_consensus.cc:348] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.426133 19024 raft_consensus.cc:374] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:07.426273 19024 raft_consensus.cc:729] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b23d578b1e54bb0811dc3237b22f2b6, State: Initialized, Role: FOLLOWER
I20260620 07:04:07.427690 19024 consensus_queue.cc:260] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.428283 19023 ts_tablet_manager.cc:1428] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s
I20260620 07:04:07.436928 19025 tablet_bootstrap.cc:492] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: No bootstrap required, opened a new log
I20260620 07:04:07.437239 19025 ts_tablet_manager.cc:1397] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.014s
I20260620 07:04:07.438200 18846 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
I20260620 07:04:07.438891 19024 ts_tablet_manager.cc:1428] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Time spent starting tablet: real 0.025s user 0.020s sys 0.001s
I20260620 07:04:07.445709 19025 raft_consensus.cc:348] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.446076 19025 raft_consensus.cc:374] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:07.446156 19025 raft_consensus.cc:729] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43b31c9d86049bb95e2bdc05dc28e7d, State: Initialized, Role: FOLLOWER
I20260620 07:04:07.446692 19025 consensus_queue.cc:260] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.447784 18987 heartbeater.cc:503] Master 127.11.81.254:39281 was elected leader, sending a full tablet report...
I20260620 07:04:07.448374 19025 ts_tablet_manager.cc:1428] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260620 07:04:07.462486 18702 tablet.cc:2369] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:04:07.478327 18989 tablet.cc:2369] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:04:07.534300 19030 raft_consensus.cc:482] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:07.534583 19030 raft_consensus.cc:504] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.536352 19030 leader_election.cc:288] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491), b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:07.542520 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.536874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:56599 (local address 127.11.81.193:41491)
0620 07:04:07.537076 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.537081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.537324 (+ 243us) server_negotiation.cc:407] Connection header received
0620 07:04:07.537515 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.537520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.537579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.537656 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:07.538126 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.538831 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.540932 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.541675 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.541830 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.541905 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.542168 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.542285 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:04:07.542730 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.536804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:51903 (local address 127.11.81.195:42437)
0620 07:04:07.537017 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.537024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.537689 (+ 665us) server_negotiation.cc:407] Connection header received
0620 07:04:07.537823 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.537828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.537885 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.537954 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:07.538448 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.539126 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.541123 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.541912 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.542053 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.542123 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.542387 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.542506 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:04:07.542863 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.536607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42437 (local address 127.11.81.194:51903)
0620 07:04:07.537561 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.537579 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.537701 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.538001 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.538005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.538028 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:07.538307 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.538315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.539257 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.539262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.540985 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.540996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.542122 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.542129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.542255 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.542276 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.542350 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.542471 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":422,"thread_start_us":277,"threads_started":1}
I20260620 07:04:07.542881 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.536607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41491 (local address 127.11.81.194:56599)
0620 07:04:07.537214 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:07.537233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:07.537383 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:07.537697 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:07.537702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:07.537726 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:07.537997 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.538005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.538955 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.538960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.540797 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:07.540809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.541861 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.541867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:07.542014 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.542037 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:04:07.542131 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:04:07.542257 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":21,"thread_start_us":300,"threads_started":1}
I20260620 07:04:07.543776 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01876c8e3004e7990857b4cd1d6e97d" candidate_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" is_pre_election: true
I20260620 07:04:07.543824 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01876c8e3004e7990857b4cd1d6e97d" candidate_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc863e8db7d44ce81aedb7094c723e9" is_pre_election: true
I20260620 07:04:07.544246 18654 raft_consensus.cc:2454] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b23d578b1e54bb0811dc3237b22f2b6 in term 0.
I20260620 07:04:07.544245 18941 raft_consensus.cc:2454] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b23d578b1e54bb0811dc3237b22f2b6 in term 0.
I20260620 07:04:07.544824 18735 leader_election.cc:302] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc863e8db7d44ce81aedb7094c723e9, 9b23d578b1e54bb0811dc3237b22f2b6; no voters:
I20260620 07:04:07.545188 19030 raft_consensus.cc:2781] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:07.545337 19030 raft_consensus.cc:482] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:07.545431 19030 raft_consensus.cc:3037] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:07.547618 19030 raft_consensus.cc:504] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.548216 19030 leader_election.cc:288] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [CANDIDATE]: Term 1 election: Requested vote from peers 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491), b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437)
I20260620 07:04:07.548693 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01876c8e3004e7990857b4cd1d6e97d" candidate_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc863e8db7d44ce81aedb7094c723e9"
I20260620 07:04:07.548789 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01876c8e3004e7990857b4cd1d6e97d" candidate_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d"
I20260620 07:04:07.548916 18654 raft_consensus.cc:3037] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:07.548983 18941 raft_consensus.cc:3037] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:07.552076 18654 raft_consensus.cc:2454] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b23d578b1e54bb0811dc3237b22f2b6 in term 1.
I20260620 07:04:07.552076 18941 raft_consensus.cc:2454] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b23d578b1e54bb0811dc3237b22f2b6 in term 1.
I20260620 07:04:07.552551 18735 leader_election.cc:302] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cc863e8db7d44ce81aedb7094c723e9, 9b23d578b1e54bb0811dc3237b22f2b6, b43b31c9d86049bb95e2bdc05dc28e7d; no voters:
I20260620 07:04:07.552897 19030 raft_consensus.cc:2781] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:07.553335 19030 raft_consensus.cc:686] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 9b23d578b1e54bb0811dc3237b22f2b6, State: Running, Role: LEADER
I20260620 07:04:07.554177 19030 consensus_queue.cc:237] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } }
I20260620 07:04:07.560175 17729 catalog_manager.cc:5495] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } health_report { overall_health: UNKNOWN } } }
W20260620 07:04:07.571689 18848 tablet.cc:2369] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:04:07.598188 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.592929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.11.81.194:35789)
0620 07:04:07.593151 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.593157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.593251 (+ 94us) server_negotiation.cc:407] Connection header received
0620 07:04:07.593492 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.593496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.593575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.593717 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:07.594075 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.594767 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.596133 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.596519 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.596627 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.597097 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:07.597606 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:07.597741 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.597839 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.597918 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:04:07.606474 18654 raft_consensus.cc:1261] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Refusing update from remote peer 9b23d578b1e54bb0811dc3237b22f2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:07.606474 18941 raft_consensus.cc:1261] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Refusing update from remote peer 9b23d578b1e54bb0811dc3237b22f2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:07.607560 19034 consensus_queue.cc:1035] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:07.607939 19030 consensus_queue.cc:1035] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:07.624984 19037 mvcc.cc:205] Tried to move back new op lower bound from 7298822338985934848 to 7298822338786131968. Current Snapshot: MvccSnapshot[applied={T|T < 7298822338985934848 or (T in {7298822338985934848})}]
I20260620 07:04:07.640014 17729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:07.640311 17729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:07.646672 17729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8f6c007f42c44dfb4348a1ab0361b0d: Sending DeleteTablet for 3 replicas of tablet b01876c8e3004e7990857b4cd1d6e97d
I20260620 07:04:07.647732 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet b01876c8e3004e7990857b4cd1d6e97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:07 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:47308
I20260620 07:04:07.648494 18780 tablet_service.cc:1514] Processing DeleteTablet for tablet b01876c8e3004e7990857b4cd1d6e97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:07 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:47118
I20260620 07:04:07.649461 19047 tablet_replica.cc:330] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: stopping tablet replica
I20260620 07:04:07.649899 19047 raft_consensus.cc:2229] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:07.649878 18921 tablet_service.cc:1514] Processing DeleteTablet for tablet b01876c8e3004e7990857b4cd1d6e97d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:07 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:39738
I20260620 07:04:07.650440 19047 raft_consensus.cc:2258] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:07.651407 19048 tablet_replica.cc:330] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: stopping tablet replica
I20260620 07:04:07.651882 19048 raft_consensus.cc:2229] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:07.652632 19047 ts_tablet_manager.cc:1905] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:07.652642 19048 raft_consensus.cc:2258] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:07.654956 19048 ts_tablet_manager.cc:1905] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:07.660499 19047 ts_tablet_manager.cc:1918] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:07.660701 19047 log.cc:1196] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/b01876c8e3004e7990857b4cd1d6e97d
I20260620 07:04:07.661307 19047 ts_tablet_manager.cc:1939] T b01876c8e3004e7990857b4cd1d6e97d P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting consensus metadata
I20260620 07:04:07.662237 19048 ts_tablet_manager.cc:1918] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:07.662490 19048 log.cc:1196] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/b01876c8e3004e7990857b4cd1d6e97d
I20260620 07:04:07.663240 17714 catalog_manager.cc:4841] TS 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789): tablet b01876c8e3004e7990857b4cd1d6e97d (table test-table [id=8ec8fb27f71746e19094ea6421f0d470]) successfully deleted
I20260620 07:04:07.663631 19048 ts_tablet_manager.cc:1939] T b01876c8e3004e7990857b4cd1d6e97d P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting consensus metadata
I20260620 07:04:07.664614 19046 tablet_replica.cc:330] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: stopping tablet replica
I20260620 07:04:07.665112 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 17697
I20260620 07:04:07.665081 19046 raft_consensus.cc:2229] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:07.665596 19046 raft_consensus.cc:2258] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:07.666249 17759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:07.668208 19046 ts_tablet_manager.cc:1905] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:07.675853 19046 ts_tablet_manager.cc:1918] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:07.676056 19046 log.cc:1196] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/b01876c8e3004e7990857b4cd1d6e97d
I20260620 07:04:07.676620 19046 ts_tablet_manager.cc:1939] T b01876c8e3004e7990857b4cd1d6e97d P 1cc863e8db7d44ce81aedb7094c723e9: Deleting consensus metadata
I20260620 07:04:07.692086 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.651906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.11.81.253:36195)
0620 07:04:07.652091 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.652097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.663930 (+ 11833us) server_negotiation.cc:407] Connection header received
0620 07:04:07.664164 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.664169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.664259 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.664355 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.668251 (+ 3896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.669530 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.672500 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.675500 (+ 3000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.675647 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.680903 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.680912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.680917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.680973 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.687862 (+ 6889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.689020 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.689027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.689031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.689207 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.689727 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.689733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.689736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.690189 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.690435 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.691527 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.691761 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:04:07.706431 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.664084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.11.81.252:38777)
0620 07:04:07.664635 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.664641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.691764 (+ 27123us) server_negotiation.cc:407] Connection header received
0620 07:04:07.692000 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.692005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.692090 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.692161 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.694153 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.694849 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.696161 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.696575 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.696687 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.699097 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.699101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.699112 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.699147 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.702942 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.703873 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.703879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.703882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.704038 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.704523 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.704529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.704533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.704920 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.705228 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.705783 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.706017 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260620 07:04:07.826545 17713 catalog_manager.cc:4841] TS b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437): tablet b01876c8e3004e7990857b4cd1d6e97d (table test-table [id=8ec8fb27f71746e19094ea6421f0d470]) successfully deleted
W20260620 07:04:07.826788 17713 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.81.195:42437, user_credentials={real_user=slave}} blocked reactor thread for 148965us
I20260620 07:04:07.827531 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 17697
I20260620 07:04:07.881954 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.861890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.11.81.252:38777)
0620 07:04:07.862082 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.862088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.866044 (+ 3956us) server_negotiation.cc:407] Connection header received
0620 07:04:07.866292 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.866298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.866386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.866456 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.868920 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.870056 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.872555 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.873172 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.873279 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.876371 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.876375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.876379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.876411 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.878904 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.879578 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.879581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.879584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.879736 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.880497 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.880502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.880505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.880836 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.881034 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.881389 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.881598 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:04:07.882433 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:07.862784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.11.81.253:36195)
0620 07:04:07.862981 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:07.862987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:07.865681 (+ 2694us) server_negotiation.cc:407] Connection header received
0620 07:04:07.866131 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:07.866138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:07.866222 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:07.866292 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:07.868395 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.869537 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.871804 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:07.872450 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:07.872544 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:07.876350 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:07.876356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:07.876360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:07.876416 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:07.879535 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.880207 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.880211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.880213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.880343 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:07.880773 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:07.880779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:07.880782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:07.881143 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:07.881370 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:07.881914 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 07:04:07.882127 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:04:08.632218 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:08.631511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:51371)
0620 07:04:08.631929 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:08.632041 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":190,"threads_started":1}
W20260620 07:04:08.632563 18846 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:39281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:08.638586 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:08.637728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:39175)
0620 07:04:08.638204 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:08.638339 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":236,"threads_started":1}
W20260620 07:04:08.639098 18987 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:39281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:08.640106 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:08.639379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:48927)
0620 07:04:08.639832 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:08.639954 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":247,"threads_started":1}
W20260620 07:04:08.640479 18700 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:39281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:09.147585 19061 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f6c007f42c44dfb4348a1ab0361b0d)
I20260620 07:04:09.147866 19061 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } }
I20260620 07:04:09.149003 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f6c007f42c44dfb4348a1ab0361b0d)
I20260620 07:04:09.149217 19061 leader_election.cc:288] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281), 4e7fbaddb09d4f92adcac937865aeccd (127.11.81.252:38777)
I20260620 07:04:09.149173 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } }
I20260620 07:04:09.150069 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281), f62d5048314542a0b09a31f3311198f4 (127.11.81.253:36195)
I20260620 07:04:09.151019 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.149367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48426)
0620 07:04:09.150583 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.150773 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1071,"threads_started":1}
I20260620 07:04:09.152333 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.150144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48436)
0620 07:04:09.151857 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.152050 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1601,"threads_started":1}
W20260620 07:04:09.153455 18505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
W20260620 07:04:09.153728 18312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:09.161187 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.150866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:36195 (local address 127.0.0.1:43910)
0620 07:04:09.152498 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.152534 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.152658 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.153541 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.153546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.153573 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.153807 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.153816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.155886 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.155892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.158251 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.158270 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.160234 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.160240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.160470 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.160508 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.160656 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.160822 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"thread_start_us":756,"threads_started":1}
I20260620 07:04:09.161407 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.160652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48444)
0620 07:04:09.160937 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.161109 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260620 07:04:09.161618 18505 leader_election.cc:334] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281): Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:09.161943 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.150607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.11.81.253:36195)
0620 07:04:09.153122 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.153128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.153147 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:04:09.153210 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.153214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.153292 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.153394 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.154109 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.155631 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.158453 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.159863 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.159997 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.160102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.161422 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.161600 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2419,"thread_start_us":2248,"threads_started":1}
I20260620 07:04:09.162505 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.161597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48458)
0620 07:04:09.161821 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.162025 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 07:04:09.162438 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e7fbaddb09d4f92adcac937865aeccd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f62d5048314542a0b09a31f3311198f4" is_pre_election: true
W20260620 07:04:09.162669 18312 leader_election.cc:334] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281): Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:09.162822 18337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7fbaddb09d4f92adcac937865aeccd in term 1.
I20260620 07:04:09.163303 18507 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e7fbaddb09d4f92adcac937865aeccd, f62d5048314542a0b09a31f3311198f4; no voters: b8f6c007f42c44dfb4348a1ab0361b0d
I20260620 07:04:09.163661 19062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 07:04:09.163775 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Starting leader election (detected failure of leader b8f6c007f42c44dfb4348a1ab0361b0d)
I20260620 07:04:09.163868 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:04:09.165999 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } }
I20260620 07:04:09.166616 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 election: Requested vote from peers b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281), f62d5048314542a0b09a31f3311198f4 (127.11.81.253:36195)
I20260620 07:04:09.167086 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.166701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48460)
0620 07:04:09.166847 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.166941 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:04:09.167212 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e7fbaddb09d4f92adcac937865aeccd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f62d5048314542a0b09a31f3311198f4"
I20260620 07:04:09.167485 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.149477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:38777 (local address 127.0.0.1:46314)
0620 07:04:09.151190 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.151209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.151386 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.162374 (+ 10988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.162378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.162406 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.162665 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.162677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.164123 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.164129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.165358 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.165370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.166644 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.166651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.166829 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.166865 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.166949 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.167106 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"thread_start_us":891,"threads_started":1}
I20260620 07:04:09.167534 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.149723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.11.81.252:38777)
0620 07:04:09.162079 (+ 12356us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.162085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.162102 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:04:09.162154 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.162158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.162224 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.162324 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.162853 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.164005 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.165537 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.166439 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.166556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.166669 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.167005 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.167200 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12250,"thread_start_us":12457,"threads_started":1}
I20260620 07:04:09.167490 18337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:04:09.168316 18531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f62d5048314542a0b09a31f3311198f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e7fbaddb09d4f92adcac937865aeccd" is_pre_election: true
I20260620 07:04:09.168494 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.168106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48470)
0620 07:04:09.168232 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.168318 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 07:04:09.168740 18531 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f62d5048314542a0b09a31f3311198f4 in current term 2: Already voted for candidate 4e7fbaddb09d4f92adcac937865aeccd in this term.
W20260620 07:04:09.168705 18505 leader_election.cc:334] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281): Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
I20260620 07:04:09.169332 18315 leader_election.cc:302] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f62d5048314542a0b09a31f3311198f4; no voters: 4e7fbaddb09d4f92adcac937865aeccd, b8f6c007f42c44dfb4348a1ab0361b0d
I20260620 07:04:09.170290 18337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7fbaddb09d4f92adcac937865aeccd in term 2.
I20260620 07:04:09.170442 19061 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:04:09.170634 18507 leader_election.cc:302] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e7fbaddb09d4f92adcac937865aeccd, f62d5048314542a0b09a31f3311198f4; no voters: b8f6c007f42c44dfb4348a1ab0361b0d
I20260620 07:04:09.170877 19062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:04:09.171335 19062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [term 2 LEADER]: Becoming Leader. State: Replica: 4e7fbaddb09d4f92adcac937865aeccd, State: Running, Role: LEADER
I20260620 07:04:09.171744 19062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } }
I20260620 07:04:09.173373 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7fbaddb09d4f92adcac937865aeccd. Latest consensus state: current_term: 2 leader_uuid: "4e7fbaddb09d4f92adcac937865aeccd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:09.173792 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:09.174448 19073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:09.177908 19073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:09.178689 19073 catalog_manager.cc:1259] Loaded cluster ID: 2aa9a61faac244f686222bd4dfe7e779
I20260620 07:04:09.178761 19073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:09.179878 19073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:09.180646 19073 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Loaded TSK: 0
I20260620 07:04:09.181458 19073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:04:09.216485 18521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:09.223654 18521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:09.228049 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.227644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48490)
0620 07:04:09.227802 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.227900 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:04:09.228242 18337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [term 2 FOLLOWER]: Refusing update from remote peer 4e7fbaddb09d4f92adcac937865aeccd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 07:04:09.229050 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.228665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48504)
0620 07:04:09.228794 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.228878 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 07:04:09.228979 19071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 07:04:09.229262 18505 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd -> Peer b8f6c007f42c44dfb4348a1ab0361b0d (127.11.81.254:39281): Couldn't send request to peer b8f6c007f42c44dfb4348a1ab0361b0d. Status: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:04:09.231801 19061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e7fbaddb09d4f92adcac937865aeccd. Latest consensus state: current_term: 2 leader_uuid: "4e7fbaddb09d4f92adcac937865aeccd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:09.232087 19061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:04:09.233903 19061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4e7fbaddb09d4f92adcac937865aeccd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:09.234210 19061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62d5048314542a0b09a31f3311198f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:04:09.240310 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4e7fbaddb09d4f92adcac937865aeccd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b8f6c007f42c44dfb4348a1ab0361b0d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39281 } } peers { permanent_uuid: "f62d5048314542a0b09a31f3311198f4" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 36195 } attrs { promote: false } } peers { permanent_uuid: "4e7fbaddb09d4f92adcac937865aeccd" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 38777 } attrs { promote: false } } }
I20260620 07:04:09.240810 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:09.257419 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.249597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.11.81.195:42437)
0620 07:04:09.250012 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.250017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.250033 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:04:09.250103 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.250107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.250169 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.250715 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.250894 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.252037 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.255081 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.256365 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.256841 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.257010 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.257067 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.257175 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":209,"threads_started":1}
I20260620 07:04:09.257534 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.249366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42437 (local address 127.0.0.1:39750)
0620 07:04:09.249503 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.249520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.249608 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.250307 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.250315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.250340 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.250563 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.250580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.252190 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.252196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.253518 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.253531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.256541 (+ 3010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.256547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.256878 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.256922 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.257015 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.257184 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":62}
I20260620 07:04:09.258385 18921 tablet_service.cc:1467] Processing CreateTablet for tablet 85e151566b2f4dc7b640b70fae40158f (DEFAULT_TABLE table=test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:09.258462 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.248967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35789 (local address 127.0.0.1:47130)
0620 07:04:09.249118 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.249138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.249254 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.250324 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.250327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.250368 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.250674 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.250684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.252190 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.252196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.254850 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.254862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.256584 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.256590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.256813 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.257893 (+ 1080us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.257987 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.258133 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2411,"spinlock_wait_cycles":28928}
I20260620 07:04:09.258819 18921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e151566b2f4dc7b640b70fae40158f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:09.263967 19084 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Bootstrap starting.
I20260620 07:04:09.264321 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.249530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.11.81.194:35789)
0620 07:04:09.249986 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.249992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.250010 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:04:09.250079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.250085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.250153 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.250796 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.250961 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.252004 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.255021 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.256365 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.256875 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.257020 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.263859 (+ 6839us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.264043 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":796,"threads_started":1}
I20260620 07:04:09.265074 18780 tablet_service.cc:1467] Processing CreateTablet for tablet 85e151566b2f4dc7b640b70fae40158f (DEFAULT_TABLE table=test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:09.265540 18780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e151566b2f4dc7b640b70fae40158f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:09.265785 19084 tablet_bootstrap.cc:654] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:09.267594 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.250219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.11.81.193:41491)
0620 07:04:09.261326 (+ 11107us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.261331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.261347 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:04:09.261406 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.261409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.261476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.261568 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.262079 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.262771 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.264954 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.266108 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.266207 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.266310 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.267136 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.267270 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11005,"thread_start_us":11446,"threads_started":1}
I20260620 07:04:09.267633 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.249879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41491 (local address 127.0.0.1:47324)
0620 07:04:09.250376 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.250393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.250521 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.261607 (+ 11086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.261611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.261638 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.261919 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.261928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.262891 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.262897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.264690 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.264706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.266493 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.266499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.266664 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.267002 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.267084 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.267218 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":2432,"thread_start_us":526,"threads_started":1}
I20260620 07:04:09.268397 19084 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: No bootstrap required, opened a new log
I20260620 07:04:09.268515 19084 ts_tablet_manager.cc:1397] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:04:09.268462 18634 tablet_service.cc:1467] Processing CreateTablet for tablet 85e151566b2f4dc7b640b70fae40158f (DEFAULT_TABLE table=test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:09.268855 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e151566b2f4dc7b640b70fae40158f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:09.269085 19084 raft_consensus.cc:348] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.269253 19084 raft_consensus.cc:374] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:09.269306 19084 raft_consensus.cc:729] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43b31c9d86049bb95e2bdc05dc28e7d, State: Initialized, Role: FOLLOWER
I20260620 07:04:09.269544 19084 consensus_queue.cc:260] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.270314 19084 ts_tablet_manager.cc:1428] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:09.273952 18986 heartbeater.cc:503] Master 127.11.81.252:38777 was elected leader, sending a full tablet report...
I20260620 07:04:09.280057 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.270821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:56387)
0620 07:04:09.279734 (+ 8913us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.279856 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8791,"thread_start_us":8679,"threads_started":1}
I20260620 07:04:09.280791 19088 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Bootstrap starting.
I20260620 07:04:09.281167 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.280766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:56651)
0620 07:04:09.280957 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.281039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:04:09.282469 19088 tablet_bootstrap.cc:654] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:09.283733 19089 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Bootstrap starting.
I20260620 07:04:09.284547 19088 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: No bootstrap required, opened a new log
I20260620 07:04:09.284688 19088 ts_tablet_manager.cc:1397] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 07:04:09.285156 19089 tablet_bootstrap.cc:654] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:09.285333 19088 raft_consensus.cc:348] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.285527 19088 raft_consensus.cc:374] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:09.285614 19088 raft_consensus.cc:729] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b23d578b1e54bb0811dc3237b22f2b6, State: Initialized, Role: FOLLOWER
I20260620 07:04:09.285822 19088 consensus_queue.cc:260] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.286509 19088 ts_tablet_manager.cc:1428] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:04:09.287304 19089 tablet_bootstrap.cc:492] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: No bootstrap required, opened a new log
I20260620 07:04:09.287460 19089 ts_tablet_manager.cc:1397] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:04:09.288249 19089 raft_consensus.cc:348] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.288514 19089 raft_consensus.cc:374] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:09.288611 19089 raft_consensus.cc:729] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc863e8db7d44ce81aedb7094c723e9, State: Initialized, Role: FOLLOWER
I20260620 07:04:09.288641 18845 heartbeater.cc:503] Master 127.11.81.252:38777 was elected leader, sending a full tablet report...
I20260620 07:04:09.288846 19089 consensus_queue.cc:260] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.289710 19089 ts_tablet_manager.cc:1428] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:09.292223 18699 heartbeater.cc:503] Master 127.11.81.252:38777 was elected leader, sending a full tablet report...
I20260620 07:04:09.298427 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.290235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:41839)
0620 07:04:09.298086 (+ 7851us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.298224 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7732,"thread_start_us":7580,"threads_started":1}
I20260620 07:04:09.299467 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.299058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:40745)
0620 07:04:09.299222 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.299337 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:04:09.300621 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.286855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:41679)
0620 07:04:09.300355 (+ 13500us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.300468 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13395,"threads_started":1}
I20260620 07:04:09.301750 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.301349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:52487)
0620 07:04:09.301555 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.301632 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 07:04:09.381970 19086 raft_consensus.cc:482] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:09.382196 19086 raft_consensus.cc:504] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.383893 19086 leader_election.cc:288] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789), 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:09.389961 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.384297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:52477 (local address 127.11.81.193:41491)
0620 07:04:09.384517 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.384522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.384545 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:04:09.384704 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.384708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.384765 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.384837 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.385336 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.386251 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.388116 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.388936 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.389089 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.389152 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.389569 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.389696 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:04:09.390056 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.384413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33441 (local address 127.11.81.194:35789)
0620 07:04:09.384600 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.384605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.385047 (+ 442us) server_negotiation.cc:407] Connection header received
0620 07:04:09.385185 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.385189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.385260 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.385340 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:09.385798 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.386600 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.388278 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.389113 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.389265 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.389336 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.389668 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.389804 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:04:09.390098 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.384156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41491 (local address 127.11.81.195:52477)
0620 07:04:09.384360 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.384380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.384532 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.384892 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.384897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.384938 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.385186 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.385197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.386379 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.386385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.387963 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.387975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.389122 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.389128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.389318 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.389351 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.389551 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.389763 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":61}
I20260620 07:04:09.390125 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.384156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35789 (local address 127.11.81.195:33441)
0620 07:04:09.384945 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.384962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:09.385073 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:09.385369 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:09.385373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:09.385397 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:09.385673 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.385684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.386733 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.386737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.388123 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:09.388137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.389279 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.389283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:09.389448 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.389474 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:04:09.389559 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:04:09.389689 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":222,"thread_start_us":322,"threads_started":1}
I20260620 07:04:09.390731 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e151566b2f4dc7b640b70fae40158f" candidate_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc863e8db7d44ce81aedb7094c723e9" is_pre_election: true
I20260620 07:04:09.391001 18654 raft_consensus.cc:2454] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43b31c9d86049bb95e2bdc05dc28e7d in term 0.
I20260620 07:04:09.390914 18800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e151566b2f4dc7b640b70fae40158f" candidate_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" is_pre_election: true
I20260620 07:04:09.391325 18800 raft_consensus.cc:2454] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43b31c9d86049bb95e2bdc05dc28e7d in term 0.
I20260620 07:04:09.391544 18876 leader_election.cc:302] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc863e8db7d44ce81aedb7094c723e9, b43b31c9d86049bb95e2bdc05dc28e7d; no voters:
I20260620 07:04:09.391938 19086 raft_consensus.cc:2781] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:09.392050 19086 raft_consensus.cc:482] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:09.392144 19086 raft_consensus.cc:3037] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:09.394232 19086 raft_consensus.cc:504] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.394794 19086 leader_election.cc:288] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [CANDIDATE]: Term 1 election: Requested vote from peers 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789), 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491)
I20260620 07:04:09.395272 18800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e151566b2f4dc7b640b70fae40158f" candidate_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b23d578b1e54bb0811dc3237b22f2b6"
I20260620 07:04:09.395272 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e151566b2f4dc7b640b70fae40158f" candidate_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc863e8db7d44ce81aedb7094c723e9"
I20260620 07:04:09.395496 18654 raft_consensus.cc:3037] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:09.395496 18800 raft_consensus.cc:3037] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:09.398377 18654 raft_consensus.cc:2454] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43b31c9d86049bb95e2bdc05dc28e7d in term 1.
I20260620 07:04:09.398377 18800 raft_consensus.cc:2454] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43b31c9d86049bb95e2bdc05dc28e7d in term 1.
I20260620 07:04:09.398842 18875 leader_election.cc:302] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b23d578b1e54bb0811dc3237b22f2b6, b43b31c9d86049bb95e2bdc05dc28e7d; no voters:
I20260620 07:04:09.399145 19086 raft_consensus.cc:2781] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:09.399585 19086 raft_consensus.cc:686] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 LEADER]: Becoming Leader. State: Replica: b43b31c9d86049bb95e2bdc05dc28e7d, State: Running, Role: LEADER
I20260620 07:04:09.400151 19086 consensus_queue.cc:237] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } }
I20260620 07:04:09.400527 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.400113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:57295)
0620 07:04:09.400275 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.400373 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:04:09.403548 18520 catalog_manager.cc:5495] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195). New cstate: current_term: 1 leader_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43b31c9d86049bb95e2bdc05dc28e7d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 } health_report { overall_health: UNKNOWN } } }
I20260620 07:04:09.453541 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.448577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.11.81.195:42437)
0620 07:04:09.448764 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.448769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.448789 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:04:09.448966 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.448970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.449066 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.449190 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:09.449535 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.450253 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.451537 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.451945 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.452060 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.452457 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:09.452974 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:09.453109 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.453216 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.453295 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:04:09.457697 18654 raft_consensus.cc:1261] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Refusing update from remote peer b43b31c9d86049bb95e2bdc05dc28e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:09.458069 18800 raft_consensus.cc:1261] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Refusing update from remote peer b43b31c9d86049bb95e2bdc05dc28e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:09.458395 19097 consensus_queue.cc:1035] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc863e8db7d44ce81aedb7094c723e9" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:09.458751 19086 consensus_queue.cc:1035] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b23d578b1e54bb0811dc3237b22f2b6" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:09.461537 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.460959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:35165)
0620 07:04:09.461212 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.461312 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 07:04:09.466023 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.460911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:36563)
0620 07:04:09.465655 (+ 4744us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.465796 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4604}
I20260620 07:04:09.472072 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.471557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.194:41461)
0620 07:04:09.471803 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.471900 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 07:04:09.476728 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.475928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.193:44069)
0620 07:04:09.476459 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.476556 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
I20260620 07:04:09.477800 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.475254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.11.81.195:55085)
0620 07:04:09.477394 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.477632 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 07:04:09.489116 18521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:09.489398 18521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:09.495179 18521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e7fbaddb09d4f92adcac937865aeccd: Sending DeleteTablet for 3 replicas of tablet 85e151566b2f4dc7b640b70fae40158f
I20260620 07:04:09.496217 18780 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e151566b2f4dc7b640b70fae40158f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:09 UTC) from {username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:47130
I20260620 07:04:09.496417 18921 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e151566b2f4dc7b640b70fae40158f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:09 UTC) from {username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:39750
I20260620 07:04:09.496873 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e151566b2f4dc7b640b70fae40158f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:09 UTC) from {username='oryx', principal='oryx/127.11.81.252@KRBTEST.COM'} at 127.0.0.1:47324
I20260620 07:04:09.497224 19111 tablet_replica.cc:330] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: stopping tablet replica
I20260620 07:04:09.497256 19110 tablet_replica.cc:330] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: stopping tablet replica
I20260620 07:04:09.497439 19110 raft_consensus.cc:2229] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:09.497491 19111 raft_consensus.cc:2229] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:09.497771 19110 raft_consensus.cc:2258] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:09.498649 19111 raft_consensus.cc:2258] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:09.499110 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18499
I20260620 07:04:09.498850 19110 ts_tablet_manager.cc:1905] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:09.506430 19110 ts_tablet_manager.cc:1918] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:09.506590 19110 log.cc:1196] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/85e151566b2f4dc7b640b70fae40158f
I20260620 07:04:09.507019 19110 ts_tablet_manager.cc:1939] T 85e151566b2f4dc7b640b70fae40158f P 9b23d578b1e54bb0811dc3237b22f2b6: Deleting consensus metadata
I20260620 07:04:09.507591 19111 ts_tablet_manager.cc:1905] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:09.508913 18506 catalog_manager.cc:4841] TS 9b23d578b1e54bb0811dc3237b22f2b6 (127.11.81.194:35789): tablet 85e151566b2f4dc7b640b70fae40158f (table test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]) successfully deleted
I20260620 07:04:09.514622 19112 tablet_replica.cc:330] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: stopping tablet replica
I20260620 07:04:09.514966 19112 raft_consensus.cc:2229] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:09.515314 19111 ts_tablet_manager.cc:1918] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:09.515473 19111 log.cc:1196] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/85e151566b2f4dc7b640b70fae40158f
I20260620 07:04:09.515393 19112 raft_consensus.cc:2258] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:09.515950 19111 ts_tablet_manager.cc:1939] T 85e151566b2f4dc7b640b70fae40158f P b43b31c9d86049bb95e2bdc05dc28e7d: Deleting consensus metadata
I20260620 07:04:09.516654 19112 ts_tablet_manager.cc:1905] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:09.517151 18505 catalog_manager.cc:4841] TS b43b31c9d86049bb95e2bdc05dc28e7d (127.11.81.195:42437): tablet 85e151566b2f4dc7b640b70fae40158f (table test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]) successfully deleted
I20260620 07:04:09.523718 19112 ts_tablet_manager.cc:1918] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:09.523954 19112 log.cc:1196] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/85e151566b2f4dc7b640b70fae40158f
I20260620 07:04:09.524334 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.500949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.11.81.193:41491)
0620 07:04:09.501127 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.501134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.507761 (+ 6627us) server_negotiation.cc:407] Connection header received
0620 07:04:09.507986 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.507993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.508108 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.508211 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:09.510238 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.511385 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.513265 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.513915 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.514014 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.517445 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:09.517465 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:09.517485 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:09.517526 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:09.520893 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.521773 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.521783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.521787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.521935 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.522357 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.522362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.522365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.522828 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:09.523108 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.523713 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.523964 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:04:09.524740 19112 ts_tablet_manager.cc:1939] T 85e151566b2f4dc7b640b70fae40158f P 1cc863e8db7d44ce81aedb7094c723e9: Deleting consensus metadata
I20260620 07:04:09.525614 18694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:09.525967 18507 catalog_manager.cc:4841] TS 1cc863e8db7d44ce81aedb7094c723e9 (127.11.81.193:41491): tablet 85e151566b2f4dc7b640b70fae40158f (table test-table [id=365af8e332e54358b3f68f9cdc6f4a5f]) successfully deleted
I20260620 07:04:09.644402 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18499
I20260620 07:04:09.665355 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18711
I20260620 07:04:09.680879 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.666141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.11.81.194:35789)
0620 07:04:09.666315 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.666325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.666351 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:04:09.666572 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.666576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.666639 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.666720 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:09.668452 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.669190 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.671091 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.671689 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.671812 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.675019 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:09.675036 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:09.675052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:09.675105 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:09.678014 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.678840 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.678847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.678850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.678984 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.679501 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.679505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.679508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.679885 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:09.680127 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.680443 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.680618 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:04:09.681517 18840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:09.748588 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.748054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48510)
0620 07:04:09.748252 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.748381 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:04:09.752545 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.752053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39281 (local address 127.0.0.1:48516)
0620 07:04:09.752244 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:09.752360 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:04:09.805578 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18711
I20260620 07:04:09.823608 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18858
I20260620 07:04:09.841177 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:09.824473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.11.81.195:42437)
0620 07:04:09.824669 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:09.824675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:09.824773 (+ 98us) server_negotiation.cc:407] Connection header received
0620 07:04:09.824959 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:09.824965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:09.825076 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:09.825190 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:09.826875 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.828069 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.829863 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:09.830530 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:09.830672 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:09.834070 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:09.834091 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:09.834115 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:09.834162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:09.838165 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.839050 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.839058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.839060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.839207 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:09.839662 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:09.839665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:09.839667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:09.840042 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:09.840270 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:09.840730 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 07:04:09.840912 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:04:09.842010 18981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:09.967473 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18858
I20260620 07:04:09.984217 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18190
I20260620 07:04:09.985338 18358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:10.100666 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18190
I20260620 07:04:10.117030 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 18291
I20260620 07:04:10.118197 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:10.238760 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 18291
2026-06-20T07:04:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9472 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:10 dist-test-slave-wvz3 krb5kdc[19122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:10 dist-test-slave-wvz3 krb5kdc[19122](info): set up 2 sockets
Jun 20 07:04:10 dist-test-slave-wvz3 krb5kdc[19122](info): commencing operation
krb5kdc: starting...
W20260620 07:04:11.260138 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:11 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab.
Entry for principal oryx/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab.
I20260620 07:04:11.479139 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36895
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab
--principal=oryx/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:11.809576 19135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:11.809899 19135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:11.810048 19135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:11.817389 19135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:11.817517 19135 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:04:11.817559 19135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:11.817621 19135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:11.817670 19135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:11.817728 19135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:11.827860 19135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:36895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.254.keytab
--principal=oryx/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:11.829141 19135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:11.830525 19135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:11.842593 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:11.842589 19143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:11.842538 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:11.843518 19135 server_base.cc:1034] running on GCE node
I20260620 07:04:11.844264 19135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:11.845636 19135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:11.846848 19135 hybrid_clock.cc:648] HybridClock initialized: now 1781939051846779 us; error 47 us; skew 500 ppm
I20260620 07:04:11.847182 19135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:11 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939051, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:11.850137 19135 init.cc:377] Logged in from keytab as oryx/127.11.81.254@KRBTEST.COM (short username oryx)
I20260620 07:04:11.851660 19135 webserver.cc:460] Webserver started at http://127.11.81.254:41399/ using document root <none> and password file <none>
I20260620 07:04:11.852362 19135 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:11.852468 19135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:11.852769 19135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:11.855741 19135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "e70eb8639c6249998b3b6e3e53fcb572"
format_stamp: "Formatted at 2026-06-20 07:04:11 on dist-test-slave-wvz3"
server_key: "715997cf8ce375e5a9d9014e94851ed3"
server_key_iv: "dde182b8e4f7e5ed4632e880a32a353d"
server_key_version: "encryptionkey@0"
I20260620 07:04:11.856505 19135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "e70eb8639c6249998b3b6e3e53fcb572"
format_stamp: "Formatted at 2026-06-20 07:04:11 on dist-test-slave-wvz3"
server_key: "715997cf8ce375e5a9d9014e94851ed3"
server_key_iv: "dde182b8e4f7e5ed4632e880a32a353d"
server_key_version: "encryptionkey@0"
I20260620 07:04:11.861568 19135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:11.865013 19150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:11.866379 19135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:11.866572 19135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "e70eb8639c6249998b3b6e3e53fcb572"
format_stamp: "Formatted at 2026-06-20 07:04:11 on dist-test-slave-wvz3"
server_key: "715997cf8ce375e5a9d9014e94851ed3"
server_key_iv: "dde182b8e4f7e5ed4632e880a32a353d"
server_key_version: "encryptionkey@0"
I20260620 07:04:11.866768 19135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:11.883677 19135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:11.887823 19135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:11.888077 19135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:11.908082 19135 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:36895
I20260620 07:04:11.908102 19211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:36895 every 8 connection(s)
I20260620 07:04:11.909395 19135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:11.912232 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19135
I20260620 07:04:11.912441 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:11.913225 11591 external_mini_cluster.cc:1428] Setting key 5b73bde5a6c95fcf83f32b64beaf34f9
I20260620 07:04:11.913635 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:11.928114 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Bootstrap starting.
Jun 20 07:04:11 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:11.931566 19212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:11.932663 19212 log.cc:824] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:11.935434 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: No bootstrap required, opened a new log
I20260620 07:04:11.936506 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:11.915224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.11.81.254:36895)
0620 07:04:11.916006 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:11.916018 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:11.916057 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:04:11.918211 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:11.918234 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:11.918789 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:11.919147 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:11.921232 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:11.922514 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:11.924180 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:11.924942 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:11.925094 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:11.929479 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:11.929487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:11.929506 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:11.929548 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:11.932515 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:11.933231 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:11.933239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:11.933245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:11.933421 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:11.933792 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:11.933796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:11.933798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:11.934324 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:11.934601 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:11.935060 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 07:04:11.935520 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260620 07:04:11.943055 19212 raft_consensus.cc:348] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } }
I20260620 07:04:11.943429 19212 raft_consensus.cc:374] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:11.943531 19212 raft_consensus.cc:729] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70eb8639c6249998b3b6e3e53fcb572, State: Initialized, Role: FOLLOWER
I20260620 07:04:11.944065 19212 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } }
I20260620 07:04:11.944262 19212 raft_consensus.cc:388] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:11.944381 19212 raft_consensus.cc:482] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:11.944540 19212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:11.946928 19212 raft_consensus.cc:504] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } }
I20260620 07:04:11.947494 19212 leader_election.cc:302] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70eb8639c6249998b3b6e3e53fcb572; no voters:
I20260620 07:04:11.948062 19212 leader_election.cc:288] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:11.948184 19217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:11.948838 19217 raft_consensus.cc:686] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [term 1 LEADER]: Becoming Leader. State: Replica: e70eb8639c6249998b3b6e3e53fcb572, State: Running, Role: LEADER
I20260620 07:04:11.949340 19217 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } }
I20260620 07:04:11.949975 19212 sys_catalog.cc:564] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:11.952230 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e70eb8639c6249998b3b6e3e53fcb572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } } }
I20260620 07:04:11.952353 19219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70eb8639c6249998b3b6e3e53fcb572. Latest consensus state: current_term: 1 leader_uuid: "e70eb8639c6249998b3b6e3e53fcb572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70eb8639c6249998b3b6e3e53fcb572" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 36895 } } }
I20260620 07:04:11.952634 19219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:11.952633 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:11.953555 19226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:11.963174 19226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:11.973342 19226 catalog_manager.cc:1347] Generated new cluster ID: 0c211f84cca04706abf67b04f9c7c035
I20260620 07:04:11.973475 19226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:11.989444 19226 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:11.991498 19226 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:12.005175 19226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Generated new TSK 0
I20260620 07:04:12.006328 19226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
I20260620 07:04:12.056870 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36895
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:12.382555 19238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:12.382844 19238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:12.382941 19238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:12.390394 19238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:12.390527 19238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:12.390611 19238 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:04:12.390691 19238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:12.400852 19238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:12.402150 19238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:12.403532 19238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:12.416730 19243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:12.416669 19244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:12.416632 19246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:12.417402 19238 server_base.cc:1034] running on GCE node
I20260620 07:04:12.418051 19238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:12.419009 19238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:12.420235 19238 hybrid_clock.cc:648] HybridClock initialized: now 1781939052420117 us; error 89 us; skew 500 ppm
I20260620 07:04:12.420536 19238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:12 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939052, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:12.423709 19238 init.cc:377] Logged in from keytab as oryx/127.11.81.193@KRBTEST.COM (short username oryx)
I20260620 07:04:12.425200 19238 webserver.cc:460] Webserver started at http://127.11.81.193:37229/ using document root <none> and password file <none>
I20260620 07:04:12.425904 19238 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:12.426017 19238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:12.426363 19238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:12.429344 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "fbf0b5230872afb9d56c4add322f8f15"
server_key_iv: "5f433c0072043b3509bd01cdd396570a"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.430155 19238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "fbf0b5230872afb9d56c4add322f8f15"
server_key_iv: "5f433c0072043b3509bd01cdd396570a"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.435484 19238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:04:12.439180 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:12.440658 19238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:12.440814 19238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "fbf0b5230872afb9d56c4add322f8f15"
server_key_iv: "5f433c0072043b3509bd01cdd396570a"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.441051 19238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:12.460033 19238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:12.464342 19238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:12.464588 19238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:12.467214 19238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:12.467327 19238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:04:12.467483 19238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:12.467545 19238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:04:12 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939052, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:12.494457 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.467940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:43457 (local address 127.11.81.254:36895)
0620 07:04:12.470792 (+ 2852us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:12.470797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:12.470819 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:04:12.473032 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:12.473039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:12.473127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:12.473248 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:12.476004 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.476992 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.479906 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.480557 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.480741 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.486444 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:12.486454 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:12.486460 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:12.486510 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:12.490246 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:12.491481 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:12.491488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:12.491492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:12.491665 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:12.492192 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:12.492199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:12.492202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:12.492611 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:12.492768 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:12.493851 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0620 07:04:12.494089 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2743,"thread_start_us":2652,"threads_started":1}
I20260620 07:04:12.495663 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.468822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.193:43457)
0620 07:04:12.469823 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:12.469882 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:12.472627 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:12.473568 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:12.473588 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:12.474628 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:12.475737 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.475773 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.477391 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.477398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.479695 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.479724 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.480880 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.480889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.481291 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.482274 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:12.482311 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:12.486080 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:12.490468 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:12.490476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:12.490495 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:12.491246 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:12.491793 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:12.491799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:12.491806 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:12.492058 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:12.492736 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:12.492745 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:12.493339 (+ 594us) client_negotiation.cc:769] Sending connection context
0620 07:04:12.493729 (+ 390us) client_negotiation.cc:240] Negotiation successful
0620 07:04:12.494321 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":386,"threads_started":1}
I20260620 07:04:12.515622 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.501667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:56737 (local address 127.11.81.254:36895)
0620 07:04:12.501893 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:12.501899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:12.502341 (+ 442us) server_negotiation.cc:407] Connection header received
0620 07:04:12.502516 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:12.502519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:12.502583 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:12.502661 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:12.504051 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.504963 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.506944 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.507381 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.507490 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.509383 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:12.509389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:12.509394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:12.509439 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:12.512545 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:12.513410 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:12.513416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:12.513420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:12.513587 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:12.514027 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:12.514033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:12.514037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:12.514448 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:12.514595 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:12.515017 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 07:04:12.515251 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:04:12.515760 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.501565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.193:56737)
0620 07:04:12.502220 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:12.502242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:12.502409 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:12.502709 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:12.502714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:12.503331 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:12.503848 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.503862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.505103 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.505109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.506782 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.506799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.507577 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.507583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.507759 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.508381 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:12.508400 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:12.509191 (+ 791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:12.512692 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:12.512697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:12.512702 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:12.513282 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:12.513715 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:12.513720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:12.513728 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:12.513913 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:12.514567 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:12.514575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:12.514777 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 07:04:12.514899 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:04:12.515269 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":314,"threads_started":1}
I20260620 07:04:12.520262 19238 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:41511
I20260620 07:04:12.520279 19373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:41511 every 8 connection(s)
I20260620 07:04:12.521471 19238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:12.523458 19374 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36895
I20260620 07:04:12.523654 19374 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:12.524269 19374 heartbeater.cc:511] Master 127.11.81.254:36895 requested a full tablet report, sending...
I20260620 07:04:12.526204 19167 ts_manager.cc:194] Registered new tserver with Master: cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511)
I20260620 07:04:12.527721 19167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:43457
I20260620 07:04:12.528230 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19238
I20260620 07:04:12.528384 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:12.529004 11591 external_mini_cluster.cc:1428] Setting key d1da9f0922588593ff4660f71805a53f
WARNING: no policy specified for oryx/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
I20260620 07:04:12.573791 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36895
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:12.894920 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:12.895213 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:12.895344 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:12.902745 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:12.902851 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:12.902961 19378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:04:12.903069 19378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:12.916055 19378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:12.917652 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:12.919267 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:12.934747 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:12.934744 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:12.935099 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:12.935443 19378 server_base.cc:1034] running on GCE node
I20260620 07:04:12.936086 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:12.937049 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:12.938319 19378 hybrid_clock.cc:648] HybridClock initialized: now 1781939052938237 us; error 58 us; skew 500 ppm
I20260620 07:04:12.938635 19378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:12 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939052, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:12.941744 19378 init.cc:377] Logged in from keytab as oryx/127.11.81.194@KRBTEST.COM (short username oryx)
I20260620 07:04:12.943251 19378 webserver.cc:460] Webserver started at http://127.11.81.194:39165/ using document root <none> and password file <none>
I20260620 07:04:12.943977 19378 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:12.944065 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:12.944375 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:12.947486 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "7e9fef8e26ea5749fc156f51afe04f98"
server_key_iv: "e68f66645705693abf1e5562dd5bbd75"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.948343 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "7e9fef8e26ea5749fc156f51afe04f98"
server_key_iv: "e68f66645705693abf1e5562dd5bbd75"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.953611 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:12.957223 19393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:12.958683 19378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:04:12.958861 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
format_stamp: "Formatted at 2026-06-20 07:04:12 on dist-test-slave-wvz3"
server_key: "7e9fef8e26ea5749fc156f51afe04f98"
server_key_iv: "e68f66645705693abf1e5562dd5bbd75"
server_key_version: "encryptionkey@0"
I20260620 07:04:12.959090 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:12.974694 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:12.978675 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:12.978935 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:12.981703 19378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:12.981792 19378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:12.981923 19378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:12.981998 19378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:04:12 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939052, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:13.004384 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.982214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46879 (local address 127.11.81.254:36895)
0620 07:04:12.982476 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:12.982483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:12.983882 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 07:04:12.985996 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:12.986002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:12.986082 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:12.986171 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:12.988608 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.989624 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.992590 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.993116 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.993249 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.998276 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:12.998282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:12.998285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:12.998324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:13.001143 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.001902 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.001908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.001912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.002039 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.002479 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.002484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.002488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.002779 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:13.002921 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:13.003780 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 07:04:13.004028 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260620 07:04:13.005872 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:12.982813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.194:46879)
0620 07:04:12.983687 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:12.983738 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:12.985658 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:12.986476 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:12.986492 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:12.987451 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:12.988401 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.988437 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.989815 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.989825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.992402 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:12.992424 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:12.993617 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:12.993622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:12.993905 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:12.994918 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:12.994946 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:12.998008 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:13.001293 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.001303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.001322 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.001759 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.002142 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.002147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.002152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.002353 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.002912 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:13.002921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:13.003347 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 07:04:13.003661 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 07:04:13.004285 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":339,"threads_started":1}
I20260620 07:04:13.026522 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.012607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:32941 (local address 127.11.81.254:36895)
0620 07:04:13.012825 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:13.012832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:13.013273 (+ 441us) server_negotiation.cc:407] Connection header received
0620 07:04:13.013428 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:13.013433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:13.013523 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:13.013639 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:13.014541 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.015972 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.017395 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.018052 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.018181 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.020565 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:13.020573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:13.020578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:13.020624 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:13.023408 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.024364 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.024371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.024375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.024524 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.024946 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.024951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.024954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.025365 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:13.025522 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:13.026093 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 07:04:13.026294 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:04:13.026899 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.012572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.194:32941)
0620 07:04:13.013185 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:13.013201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:13.013321 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:13.013659 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:13.013662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:13.014101 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:13.014407 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.014417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.016167 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.016172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.017253 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.017267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.018285 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.018292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.018494 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.019144 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:13.019168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:13.020295 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:13.023555 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.023561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.023566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.024153 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.024654 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.024660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.024664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.024832 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.025497 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:13.025502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:13.025751 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 07:04:13.025994 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 07:04:13.026398 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":315,"threads_started":1}
I20260620 07:04:13.036722 19378 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:35323
I20260620 07:04:13.036749 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:35323 every 8 connection(s)
I20260620 07:04:13.037945 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:13.040066 19513 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36895
I20260620 07:04:13.040288 19513 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:13.040944 19513 heartbeater.cc:511] Master 127.11.81.254:36895 requested a full tablet report, sending...
I20260620 07:04:13.042344 19167 ts_manager.cc:194] Registered new tserver with Master: a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323)
I20260620 07:04:13.043176 19167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:46879
I20260620 07:04:13.044307 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19378
I20260620 07:04:13.044492 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:13.045079 11591 external_mini_cluster.cc:1428] Setting key 54b5c5a40cc07d63d63f457b85ca65b2
WARNING: no policy specified for oryx/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
I20260620 07:04:13.091320 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:36895
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:13.414469 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:13.414758 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:13.414886 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:13.422494 19517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:13.422618 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:13.422740 19517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:04:13.422860 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:13.432968 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.81.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:13.434302 19517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:13.435668 19517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:13.448078 19522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:13.448122 19523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:13.448143 19525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:13.449095 19517 server_base.cc:1034] running on GCE node
I20260620 07:04:13.449674 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:13.450630 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:13.451886 19517 hybrid_clock.cc:648] HybridClock initialized: now 1781939053451794 us; error 52 us; skew 500 ppm
I20260620 07:04:13.452178 19517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:13 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939053, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:13.455202 19517 init.cc:377] Logged in from keytab as oryx/127.11.81.195@KRBTEST.COM (short username oryx)
I20260620 07:04:13.456694 19517 webserver.cc:460] Webserver started at http://127.11.81.195:37903/ using document root <none> and password file <none>
I20260620 07:04:13.457351 19517 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:13.457441 19517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:13.457732 19517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:13.460983 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
format_stamp: "Formatted at 2026-06-20 07:04:13 on dist-test-slave-wvz3"
server_key: "7ef6556ca58f8c08493350a0c72cd883"
server_key_iv: "8c313bf010587e64e2ce6583483c28bf"
server_key_version: "encryptionkey@0"
I20260620 07:04:13.461750 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
format_stamp: "Formatted at 2026-06-20 07:04:13 on dist-test-slave-wvz3"
server_key: "7ef6556ca58f8c08493350a0c72cd883"
server_key_iv: "8c313bf010587e64e2ce6583483c28bf"
server_key_version: "encryptionkey@0"
I20260620 07:04:13.466959 19517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:04:13.470499 19532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:13.472021 19517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:13.472172 19517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
format_stamp: "Formatted at 2026-06-20 07:04:13 on dist-test-slave-wvz3"
server_key: "7ef6556ca58f8c08493350a0c72cd883"
server_key_iv: "8c313bf010587e64e2ce6583483c28bf"
server_key_version: "encryptionkey@0"
I20260620 07:04:13.472441 19517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:13.492297 19517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:13.496318 19517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:13.496575 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:13.499571 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:13.499683 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:13.499822 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:13.499898 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:04:13 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939053, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:13.525784 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.500808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.195:57495)
0620 07:04:13.501739 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:13.501783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:13.503823 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:13.504642 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:13.504658 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:13.505495 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:13.506467 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.506497 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.507954 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.507960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.510470 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.510489 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.511686 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.511694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.511960 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.512820 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:13.512858 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:13.515965 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:13.520133 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.520145 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.520166 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.520925 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.521475 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.521481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.521484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.521864 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.522688 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:13.522697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:13.523259 (+ 562us) client_negotiation.cc:769] Sending connection context
0620 07:04:13.523572 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 07:04:13.524141 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":438,"threads_started":1}
I20260620 07:04:13.527189 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.500076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:57495 (local address 127.11.81.254:36895)
0620 07:04:13.500267 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:13.500274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:13.501942 (+ 1668us) server_negotiation.cc:407] Connection header received
0620 07:04:13.504119 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:13.504125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:13.504218 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:13.504320 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:13.506672 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.507787 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.510673 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.511401 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.511556 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.516227 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:13.516235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:13.516240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:13.516288 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:13.519925 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.521136 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.521141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.521145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.521325 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.522018 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.522023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.522026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.522426 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:13.522609 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:13.526607 (+ 3998us) server_negotiation.cc:299] Negotiation successful
0620 07:04:13.526858 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:04:13.530625 19374 heartbeater.cc:503] Master 127.11.81.254:36895 was elected leader, sending a full tablet report...
I20260620 07:04:13.545341 19517 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:39221
I20260620 07:04:13.545432 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:39221 every 8 connection(s)
I20260620 07:04:13.547027 19517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:13.549984 19652 heartbeater.cc:347] Connected to a master server at 127.11.81.254:36895
I20260620 07:04:13.550199 19652 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:13.550836 19652 heartbeater.cc:511] Master 127.11.81.254:36895 requested a full tablet report, sending...
I20260620 07:04:13.551606 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19517
I20260620 07:04:13.551771 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:13.552215 11591 external_mini_cluster.cc:1428] Setting key 54dc7f468fa5a62263197a8aed06f2a9
I20260620 07:04:13.552371 19167 ts_manager.cc:194] Registered new tserver with Master: 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:13.553387 19167 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:57495
I20260620 07:04:13.555050 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.536001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:56717 (local address 127.11.81.254:36895)
0620 07:04:13.536206 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:13.536213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:13.540036 (+ 3823us) server_negotiation.cc:407] Connection header received
0620 07:04:13.540244 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:13.540250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:13.540351 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:13.540472 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:13.541859 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.542924 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.544904 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.545558 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.545700 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.547883 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:13.547890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:13.547895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:13.547935 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:13.550916 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.553095 (+ 2179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.553101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.553105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.553261 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:13.553619 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:13.553623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:13.553626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:13.553964 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:13.554099 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:13.554633 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 07:04:13.554797 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:04:13.555308 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:13.535595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.11.81.195:56717)
0620 07:04:13.539900 (+ 4305us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:13.539922 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:13.540099 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:13.540532 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:13.540537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:13.541202 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:13.541682 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.541699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.543088 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.543093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.544723 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:13.544738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.545756 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.545762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:13.545936 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:13.546534 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:13.546562 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:13.547678 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:13.551020 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.551025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.551030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.552967 (+ 1937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.553379 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:13.553383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:13.553385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:13.553523 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:13.554160 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:13.554165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:13.554368 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 07:04:13.554518 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 07:04:13.554852 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4099,"thread_start_us":3918,"threads_started":1}
I20260620 07:04:13.557767 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:04:13 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939053, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:04:13 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:13 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:14.046132 19513 heartbeater.cc:503] Master 127.11.81.254:36895 was elected leader, sending a full tablet report...
I20260620 07:04:14.556682 19652 heartbeater.cc:503] Master 127.11.81.254:36895 was elected leader, sending a full tablet report...
W20260620 07:04:14.580328 19660 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.981s user 0.003s sys 0.009s
W20260620 07:04:14.581455 19262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:13.594193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.11.81.254:36895)
0620 07:04:13.594409 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:13.594416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:13.594586 (+ 170us) server_negotiation.cc:407] Connection header received
0620 07:04:13.594784 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:13.594788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:13.594854 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:13.594939 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:13.596348 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.597080 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.598361 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:13.598782 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:13.598891 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.581227 (+982336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34078: BlockingRecv error: recv got EOF from 127.0.0.1:34078 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:04:14 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939053, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:14.598526 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.585248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.11.81.254:36895)
0620 07:04:14.585500 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.585506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.585784 (+ 278us) server_negotiation.cc:407] Connection header received
0620 07:04:14.585974 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.585979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.586057 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.586142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:14.587357 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.588066 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.589263 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.589665 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.589773 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.593610 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:14.593616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:14.593619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:14.593654 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:14.596267 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:14.596881 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:14.596885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:14.596887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:14.597007 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:14.597384 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:14.597387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:14.597389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:14.597671 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:14.597948 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.598093 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.598282 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:04:14.602344 19166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:14.609174 19166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:14.631616 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.620962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35323 (local address 127.0.0.1:48016)
0620 07:04:14.621815 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:14.621867 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:14.622048 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:14.623221 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:14.623230 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:14.623262 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:14.623751 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.623762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.625934 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.625940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.628376 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.628393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.630562 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.630569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.630811 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.630874 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 07:04:14.630997 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:04:14.631199 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":432,"threads_started":1}
I20260620 07:04:14.633047 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.621192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.11.81.194:35323)
0620 07:04:14.622593 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.622601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.622628 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:14.622726 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.622734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.622988 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.623974 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:14.624185 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.625762 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.628638 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.630294 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.632218 (+ 1924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.632476 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.632554 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.632728 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":1327,"threads_started":1}
I20260620 07:04:14.639753 19443 tablet_service.cc:1467] Processing CreateTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 (DEFAULT_TABLE table=test-table [id=b57bce02bd874d158204c37f8b0ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.640494 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.621747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:39221 (local address 127.0.0.1:51792)
0620 07:04:14.631926 (+ 10179us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:14.631949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:14.632058 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:14.634909 (+ 2851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:14.634915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:14.634947 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:14.635139 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.635152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.636898 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.636903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.638323 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.638335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.639745 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.639751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.639917 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.639952 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:04:14.640019 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:04:14.640158 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10043,"mutex_wait_us":91,"thread_start_us":10676,"threads_started":1}
I20260620 07:04:14.641151 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.621895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.11.81.195:39221)
0620 07:04:14.634257 (+ 12362us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.634266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.634296 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:04:14.634398 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.634406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.634678 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.634899 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:14.635353 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.636731 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.638478 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.639578 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.640437 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.640707 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.640784 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.640936 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12073,"threads_started":1}
I20260620 07:04:14.641391 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f8c8b3f834ea3bf06a501f0f992d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.641839 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.622395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.0.0.1:56640)
0620 07:04:14.632763 (+ 10368us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:14.632787 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:14.632920 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:14.635134 (+ 2214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:14.635139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:14.635170 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:14.635587 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.635598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.637055 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.637060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.639106 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.639120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.641119 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.641124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.641281 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.641312 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:04:14.641400 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:04:14.641527 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10210,"thread_start_us":12785,"threads_started":1}
I20260620 07:04:14.643514 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.622411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.11.81.193:41511)
0620 07:04:14.634257 (+ 11846us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.634266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.634296 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:04:14.634401 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.634408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.634678 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.634899 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:14.635777 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.636872 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.639300 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.640644 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.640752 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.640946 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.643012 (+ 2066us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.643180 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11694,"thread_start_us":11061,"threads_started":1}
I20260620 07:04:14.644789 19582 tablet_service.cc:1467] Processing CreateTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 (DEFAULT_TABLE table=test-table [id=b57bce02bd874d158204c37f8b0ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.646314 19582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f8c8b3f834ea3bf06a501f0f992d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.646521 19304 tablet_service.cc:1467] Processing CreateTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 (DEFAULT_TABLE table=test-table [id=b57bce02bd874d158204c37f8b0ecb11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.647786 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40f8c8b3f834ea3bf06a501f0f992d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.656128 19677 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Bootstrap starting.
I20260620 07:04:14.660017 19678 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Bootstrap starting.
I20260620 07:04:14.660249 19677 tablet_bootstrap.cc:654] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.661392 19679 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Bootstrap starting.
I20260620 07:04:14.661767 19677 log.cc:824] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:14.663609 19678 tablet_bootstrap.cc:654] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.664635 19678 log.cc:824] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:14.664659 19679 tablet_bootstrap.cc:654] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.665237 19677 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: No bootstrap required, opened a new log
I20260620 07:04:14.665506 19677 ts_tablet_manager.cc:1397] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 07:04:14.665824 19679 log.cc:824] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:14.667608 19678 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: No bootstrap required, opened a new log
I20260620 07:04:14.667879 19678 ts_tablet_manager.cc:1397] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:04:14.669697 19679 tablet_bootstrap.cc:492] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: No bootstrap required, opened a new log
I20260620 07:04:14.669967 19679 ts_tablet_manager.cc:1397] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:04:14.675046 19678 raft_consensus.cc:348] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.675674 19678 raft_consensus.cc:374] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.675773 19678 raft_consensus.cc:729] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.675731 19677 raft_consensus.cc:348] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.676223 19677 raft_consensus.cc:374] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.676328 19677 raft_consensus.cc:729] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.676537 19678 consensus_queue.cc:260] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.677049 19677 consensus_queue.cc:260] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.678213 19678 ts_tablet_manager.cc:1428] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 07:04:14.680050 19677 ts_tablet_manager.cc:1428] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260620 07:04:14.679919 19679 raft_consensus.cc:348] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.680352 19679 raft_consensus.cc:374] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.680493 19679 raft_consensus.cc:729] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.681118 19679 consensus_queue.cc:260] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.682928 19679 ts_tablet_manager.cc:1428] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260620 07:04:14.774645 19375 tablet.cc:2369] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:04:14.790647 19514 tablet.cc:2369] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:04:14.800408 19653 tablet.cc:2369] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:04:14.841545 19684 raft_consensus.cc:482] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:14.841869 19684 raft_consensus.cc:504] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.843662 19684 leader_election.cc:288] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:14.850286 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.844256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:59937 (local address 127.11.81.195:39221)
0620 07:04:14.844438 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.844444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.845074 (+ 630us) server_negotiation.cc:407] Connection header received
0620 07:04:14.845307 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.845312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.845375 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.845463 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:14.846137 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.846805 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.848720 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.849448 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.849601 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.849675 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.849928 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.850057 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:04:14.850347 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.844022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:47381 (local address 127.11.81.193:41511)
0620 07:04:14.844230 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.844237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.845027 (+ 790us) server_negotiation.cc:407] Connection header received
0620 07:04:14.845233 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.845237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.845313 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.845406 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:14.846040 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.846776 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.848749 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.849491 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.849634 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.849705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.849974 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.850095 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:04:14.850514 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.844056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:39221 (local address 127.11.81.194:59937)
0620 07:04:14.844978 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:14.844993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:14.845129 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:14.845589 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:14.845593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:14.845621 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:14.846000 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.846008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.846926 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.846931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.848579 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.848594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.849658 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.849664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.849790 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.849813 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:04:14.849894 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:04:14.850020 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":63,"thread_start_us":514,"threads_started":1}
I20260620 07:04:14.850559 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.843962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.11.81.194:47381)
0620 07:04:14.844913 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:14.844933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:14.845079 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:14.845427 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:14.845432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:14.845456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:14.845902 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.845915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.846896 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.846902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.848598 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:14.848609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.849711 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.849718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:14.849839 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.849861 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:04:14.849938 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:04:14.850062 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":102,"thread_start_us":481,"threads_started":1}
I20260620 07:04:14.851576 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f8c8b3f834ea3bf06a501f0f992d2" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" is_pre_election: true
I20260620 07:04:14.851584 19602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f8c8b3f834ea3bf06a501f0f992d2" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" is_pre_election: true
I20260620 07:04:14.852036 19602 raft_consensus.cc:2454] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 0.
I20260620 07:04:14.852036 19324 raft_consensus.cc:2454] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 0.
I20260620 07:04:14.852640 19395 leader_election.cc:302] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:14.852995 19684 raft_consensus.cc:2781] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:14.853176 19684 raft_consensus.cc:482] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:14.853273 19684 raft_consensus.cc:3037] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:14.855422 19684 raft_consensus.cc:504] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.856021 19684 leader_election.cc:288] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 election: Requested vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:14.856415 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f8c8b3f834ea3bf06a501f0f992d2" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
I20260620 07:04:14.856460 19602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40f8c8b3f834ea3bf06a501f0f992d2" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
I20260620 07:04:14.856597 19324 raft_consensus.cc:3037] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:14.856626 19602 raft_consensus.cc:3037] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:14.858749 19602 raft_consensus.cc:2454] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 1.
I20260620 07:04:14.858767 19324 raft_consensus.cc:2454] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 1.
I20260620 07:04:14.859182 19395 leader_election.cc:302] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d330a5a1d5746d8b842e3fff5b9aae7, a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:14.859515 19684 raft_consensus.cc:2781] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:14.859946 19684 raft_consensus.cc:686] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 LEADER]: Becoming Leader. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Running, Role: LEADER
I20260620 07:04:14.860451 19684 consensus_queue.cc:237] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.864876 19166 catalog_manager.cc:5495] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20260620 07:04:14.883976 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:14.879051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.11.81.194:35323)
0620 07:04:14.879268 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:14.879272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:14.879321 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:04:14.879441 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:14.879445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:14.879529 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:14.879650 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:14.879918 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.880614 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.881934 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:14.882378 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:14.882504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:14.882826 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:14.883430 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:14.883536 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:14.883626 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 07:04:14.883711 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:04:14.892218 19324 raft_consensus.cc:1261] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Refusing update from remote peer a5a7c1fd75dc4a2b8f2e5679a1203238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:14.892251 19602 raft_consensus.cc:1261] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Refusing update from remote peer a5a7c1fd75dc4a2b8f2e5679a1203238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:14.893198 19688 consensus_queue.cc:1035] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:14.893585 19684 consensus_queue.cc:1035] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:14.903847 19691 mvcc.cc:205] Tried to move back new op lower bound from 7298822368828264448 to 7298822368711938048. Current Snapshot: MvccSnapshot[applied={T|T < 7298822368828264448}]
I20260620 07:04:14.906747 19693 mvcc.cc:205] Tried to move back new op lower bound from 7298822368828264448 to 7298822368711938048. Current Snapshot: MvccSnapshot[applied={T|T < 7298822368828264448}]
I20260620 07:04:14.912935 19692 mvcc.cc:205] Tried to move back new op lower bound from 7298822368828264448 to 7298822368711938048. Current Snapshot: MvccSnapshot[applied={T|T < 7298822368828264448}]
I20260620 07:04:14.921638 19166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:14.921905 19166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:14.925930 19166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Sending DeleteTablet for 3 replicas of tablet b40f8c8b3f834ea3bf06a501f0f992d2
I20260620 07:04:14.927168 19582 tablet_service.cc:1514] Processing DeleteTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:14 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51792
I20260620 07:04:14.927496 19443 tablet_service.cc:1514] Processing DeleteTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:14 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:48016
I20260620 07:04:14.928156 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet b40f8c8b3f834ea3bf06a501f0f992d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:14 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260620 07:04:14.929845 19167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:14.930519 19167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:14.940323 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 366140897a164e76a1b979fd441b9a85 (DEFAULT_TABLE table=test-table [id=14cf2f6107644ee78c176a9fa5a7834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.940928 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366140897a164e76a1b979fd441b9a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.941136 19703 tablet_replica.cc:330] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: stopping tablet replica
I20260620 07:04:14.941592 19703 raft_consensus.cc:2229] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:14.942085 19703 raft_consensus.cc:2258] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:14.942832 19702 tablet_replica.cc:330] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: stopping tablet replica
I20260620 07:04:14.943254 19702 raft_consensus.cc:2229] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:14.943954 19702 raft_consensus.cc:2258] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:14.945188 19703 ts_tablet_manager.cc:1905] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:14.946465 19702 ts_tablet_manager.cc:1905] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:14.948848 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 366140897a164e76a1b979fd441b9a85 (DEFAULT_TABLE table=test-table [id=14cf2f6107644ee78c176a9fa5a7834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.949275 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366140897a164e76a1b979fd441b9a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.952665 19703 ts_tablet_manager.cc:1918] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:14.952894 19703 log.cc:1196] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/b40f8c8b3f834ea3bf06a501f0f992d2
I20260620 07:04:14.953562 19703 ts_tablet_manager.cc:1939] T b40f8c8b3f834ea3bf06a501f0f992d2 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting consensus metadata
I20260620 07:04:14.955560 19679 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Bootstrap starting.
I20260620 07:04:14.955592 19152 catalog_manager.cc:4841] TS cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511): tablet b40f8c8b3f834ea3bf06a501f0f992d2 (table test-table [id=b57bce02bd874d158204c37f8b0ecb11]) successfully deleted
I20260620 07:04:14.957203 19679 tablet_bootstrap.cc:654] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.958817 19443 tablet_service.cc:1467] Processing CreateTablet for tablet 366140897a164e76a1b979fd441b9a85 (DEFAULT_TABLE table=test-table [id=14cf2f6107644ee78c176a9fa5a7834e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:14.959257 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366140897a164e76a1b979fd441b9a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:14.959311 19702 ts_tablet_manager.cc:1918] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:14.959547 19702 log.cc:1196] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/b40f8c8b3f834ea3bf06a501f0f992d2
I20260620 07:04:14.959863 19679 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: No bootstrap required, opened a new log
I20260620 07:04:14.960018 19679 ts_tablet_manager.cc:1397] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 07:04:14.960820 19679 raft_consensus.cc:348] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.961052 19679 raft_consensus.cc:374] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.961143 19679 raft_consensus.cc:729] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.961478 19702 ts_tablet_manager.cc:1939] T b40f8c8b3f834ea3bf06a501f0f992d2 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting consensus metadata
I20260620 07:04:14.961400 19679 consensus_queue.cc:260] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.962085 19679 ts_tablet_manager.cc:1428] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:14.962409 19701 tablet_replica.cc:330] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: stopping tablet replica
I20260620 07:04:14.962860 19701 raft_consensus.cc:2229] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:14.963371 19678 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Bootstrap starting.
I20260620 07:04:14.963769 19701 raft_consensus.cc:2258] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:14.964931 19678 tablet_bootstrap.cc:654] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.965919 19677 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Bootstrap starting.
I20260620 07:04:14.965919 19701 ts_tablet_manager.cc:1905] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:14.967450 19677 tablet_bootstrap.cc:654] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:14.968165 19152 catalog_manager.cc:4841] TS a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323): tablet b40f8c8b3f834ea3bf06a501f0f992d2 (table test-table [id=b57bce02bd874d158204c37f8b0ecb11]) successfully deleted
I20260620 07:04:14.969291 19678 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: No bootstrap required, opened a new log
I20260620 07:04:14.969444 19678 ts_tablet_manager.cc:1397] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 07:04:14.970264 19678 raft_consensus.cc:348] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.970489 19678 raft_consensus.cc:374] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.970572 19678 raft_consensus.cc:729] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.970824 19678 consensus_queue.cc:260] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.971439 19678 ts_tablet_manager.cc:1428] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:14.972587 19701 ts_tablet_manager.cc:1918] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:14.972747 19701 log.cc:1196] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/b40f8c8b3f834ea3bf06a501f0f992d2
I20260620 07:04:14.973290 19701 ts_tablet_manager.cc:1939] T b40f8c8b3f834ea3bf06a501f0f992d2 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting consensus metadata
I20260620 07:04:14.974184 19677 tablet_bootstrap.cc:492] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: No bootstrap required, opened a new log
I20260620 07:04:14.974355 19677 ts_tablet_manager.cc:1397] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260620 07:04:14.975263 19151 catalog_manager.cc:4841] TS 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221): tablet b40f8c8b3f834ea3bf06a501f0f992d2 (table test-table [id=b57bce02bd874d158204c37f8b0ecb11]) successfully deleted
I20260620 07:04:14.975230 19677 raft_consensus.cc:348] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.975472 19677 raft_consensus.cc:374] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:14.975577 19677 raft_consensus.cc:729] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Initialized, Role: FOLLOWER
I20260620 07:04:14.975862 19677 consensus_queue.cc:260] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:14.976426 19677 ts_tablet_manager.cc:1428] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:15.148460 19688 raft_consensus.cc:482] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:15.148701 19688 raft_consensus.cc:504] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.149492 19688 leader_election.cc:288] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.150072 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366140897a164e76a1b979fd441b9a85" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" is_pre_election: true
I20260620 07:04:15.150198 19602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366140897a164e76a1b979fd441b9a85" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" is_pre_election: true
I20260620 07:04:15.150367 19324 raft_consensus.cc:2454] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 0.
I20260620 07:04:15.150449 19602 raft_consensus.cc:2454] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 0.
I20260620 07:04:15.150852 19395 leader_election.cc:302] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.151218 19688 raft_consensus.cc:2781] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:15.151386 19688 raft_consensus.cc:482] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:15.151516 19688 raft_consensus.cc:3037] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.154878 19688 raft_consensus.cc:504] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.155757 19688 leader_election.cc:288] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 election: Requested vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.156251 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366140897a164e76a1b979fd441b9a85" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
I20260620 07:04:15.156296 19602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366140897a164e76a1b979fd441b9a85" candidate_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
I20260620 07:04:15.156456 19324 raft_consensus.cc:3037] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.156560 19602 raft_consensus.cc:3037] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.158537 19324 raft_consensus.cc:2454] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 1.
I20260620 07:04:15.159029 19395 leader_election.cc:302] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.159329 19688 raft_consensus.cc:2781] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:15.159591 19688 raft_consensus.cc:686] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 LEADER]: Becoming Leader. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Running, Role: LEADER
I20260620 07:04:15.159615 19602 raft_consensus.cc:2454] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a7c1fd75dc4a2b8f2e5679a1203238 in term 1.
I20260620 07:04:15.160120 19688 consensus_queue.cc:237] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.163034 19166 catalog_manager.cc:5495] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:04:15 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939051, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.254@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:15.224083 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.209775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.11.81.254:36895)
0620 07:04:15.210727 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.210731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.210747 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:04:15.210795 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.210798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.210859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.210946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:15.212251 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.213034 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.214250 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.214673 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.214787 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.218622 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:15.218627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:15.218631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:15.218662 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:15.221250 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:15.222149 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:15.222152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:15.222155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:15.222263 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:15.222658 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:15.222661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:15.222663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:15.222928 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:15.223047 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.223614 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.223825 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":686,"threads_started":1}
I20260620 07:04:15.224457 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.209711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:36895 (local address 127.0.0.1:34094)
0620 07:04:15.210392 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.210416 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.210590 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.211018 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.211022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.211855 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:15.212091 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.212101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.213166 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.213170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.214112 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.214123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.214833 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.214838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.215004 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.215735 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:15.215752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:15.218415 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:15.221393 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:15.221408 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:15.221416 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:15.222037 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:15.222366 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:15.222372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:15.222376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:15.222550 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:15.223074 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:15.223083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:15.223322 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.223523 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.224010 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":154,"thread_start_us":361,"threads_started":1}
I20260620 07:04:15.230499 19166 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:34094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:04:15.231933 19166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:15.239207 19304 tablet_service.cc:1467] Processing CreateTablet for tablet c140d00e912b4651839ec9d5eb84e776 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8a3c2f117884e68a6b523ea9ad38551]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:04:15.239552 19443 tablet_service.cc:1467] Processing CreateTablet for tablet c140d00e912b4651839ec9d5eb84e776 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8a3c2f117884e68a6b523ea9ad38551]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:04:15.239874 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c140d00e912b4651839ec9d5eb84e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.240119 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c140d00e912b4651839ec9d5eb84e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.247238 19678 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: Bootstrap starting.
I20260620 07:04:15.247269 19677 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: Bootstrap starting.
I20260620 07:04:15.248065 19581 tablet_service.cc:1467] Processing CreateTablet for tablet c140d00e912b4651839ec9d5eb84e776 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8a3c2f117884e68a6b523ea9ad38551]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:04:15.248579 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c140d00e912b4651839ec9d5eb84e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.248925 19678 tablet_bootstrap.cc:654] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.249082 19677 tablet_bootstrap.cc:654] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.257241 19679 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: Bootstrap starting.
I20260620 07:04:15.259205 19679 tablet_bootstrap.cc:654] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.261857 19677 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: No bootstrap required, opened a new log
I20260620 07:04:15.262077 19677 ts_tablet_manager.cc:1397] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260620 07:04:15.263024 19677 raft_consensus.cc:348] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.263407 19677 raft_consensus.cc:374] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.263494 19677 raft_consensus.cc:729] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.263851 19677 consensus_queue.cc:260] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.264377 19688 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.264679 19688 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.268899 19679 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: No bootstrap required, opened a new log
I20260620 07:04:15.268903 19677 ts_tablet_manager.cc:1428] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 07:04:15.269210 19679 ts_tablet_manager.cc:1397] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260620 07:04:15.270146 19679 raft_consensus.cc:348] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.270509 19679 raft_consensus.cc:374] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.270624 19679 raft_consensus.cc:729] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.271049 19679 consensus_queue.cc:260] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.271843 19679 ts_tablet_manager.cc:1428] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:04:15.271692 19685 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.272118 19685 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.274001 19678 tablet_bootstrap.cc:492] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: No bootstrap required, opened a new log
I20260620 07:04:15.274187 19678 ts_tablet_manager.cc:1397] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.019s
I20260620 07:04:15.274940 19678 raft_consensus.cc:348] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.275197 19678 raft_consensus.cc:374] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.275354 19678 raft_consensus.cc:729] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.275637 19678 consensus_queue.cc:260] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.276211 19678 ts_tablet_manager.cc:1428] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:04:15.276255 19683 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.276654 19683 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.280958 19683 raft_consensus.cc:482] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:15.281112 19683 raft_consensus.cc:504] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.282773 19683 leader_election.cc:288] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.291962 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.283111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35323 (local address 127.11.81.193:56299)
0620 07:04:15.283867 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.283886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.284033 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.284615 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.284620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.284648 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:15.285027 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.285039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.286499 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.286504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.288909 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.288928 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.290952 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.290960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.291170 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.291206 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.291365 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.291513 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":456,"threads_started":1}
I20260620 07:04:15.292289 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.283056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:56299 (local address 127.11.81.194:35323)
0620 07:04:15.283227 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.283233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.283993 (+ 760us) server_negotiation.cc:407] Connection header received
0620 07:04:15.284214 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.284221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.284313 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.284385 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:15.285202 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.286353 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.289140 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.290504 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.290684 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.290767 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.291394 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.291630 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:04:15.293926 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c140d00e912b4651839ec9d5eb84e776" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" is_pre_election: true
I20260620 07:04:15.294471 19463 raft_consensus.cc:2454] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 0.
I20260620 07:04:15.295167 19255 leader_election.cc:302] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.295686 19683 raft_consensus.cc:2781] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:15.295841 19683 raft_consensus.cc:482] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:15.295939 19683 raft_consensus.cc:3037] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.298528 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.284198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:44679 (local address 127.11.81.195:39221)
0620 07:04:15.284382 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.284388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.291032 (+ 6644us) server_negotiation.cc:407] Connection header received
0620 07:04:15.291165 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.291171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.291237 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.291353 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:15.292976 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.294045 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.296068 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.297233 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.297381 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.297452 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.298093 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.298239 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:04:15.299223 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.283754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:39221 (local address 127.11.81.193:44679)
0620 07:04:15.290887 (+ 7133us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.290907 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.291043 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.291524 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.291529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.291555 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:15.292013 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.292025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.294196 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.294201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.295878 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.295892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.297695 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.297704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.297886 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.297923 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.298737 (+ 814us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.298877 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6983,"thread_start_us":6828,"threads_started":1}
I20260620 07:04:15.299064 19683 raft_consensus.cc:504] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.299825 19683 leader_election.cc:288] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 election: Requested vote from peers a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.300026 19602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c140d00e912b4651839ec9d5eb84e776" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" is_pre_election: true
I20260620 07:04:15.300290 19602 raft_consensus.cc:2454] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 0.
I20260620 07:04:15.300626 19601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c140d00e912b4651839ec9d5eb84e776" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
I20260620 07:04:15.300781 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c140d00e912b4651839ec9d5eb84e776" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
I20260620 07:04:15.300879 19601 raft_consensus.cc:3037] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.300998 19463 raft_consensus.cc:3037] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.303085 19601 raft_consensus.cc:2454] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 1.
I20260620 07:04:15.303490 19254 leader_election.cc:302] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d330a5a1d5746d8b842e3fff5b9aae7, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.303757 19683 raft_consensus.cc:2781] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:15.304214 19683 raft_consensus.cc:686] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 LEADER]: Becoming Leader. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Running, Role: LEADER
I20260620 07:04:15.304317 19463 raft_consensus.cc:2454] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 1.
I20260620 07:04:15.304621 19683 consensus_queue.cc:237] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.307817 19720 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: TxnStatusTablet state changed. Reason: New leader cda7b4aa8cc542f8a0a3eae86794b956. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.308267 19720 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:04:15.308048 19166 catalog_manager.cc:5495] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193). New cstate: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20260620 07:04:15.308964 19722 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:04:15.309234 19722 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:04:15.336146 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.330431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.11.81.193:41511)
0620 07:04:15.330628 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.330635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.330800 (+ 165us) server_negotiation.cc:407] Connection header received
0620 07:04:15.331066 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.331072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.331164 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.331304 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.331661 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.332449 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.333900 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.334331 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.334445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.334891 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.335471 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.335598 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.335690 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.335787 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:04:15.336123 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.330356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.0.0.1:56654)
0620 07:04:15.330712 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.330729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.330871 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.331356 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.331360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.331374 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:04:15.331537 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.331545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.332580 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.332586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.333771 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.333784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.334473 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.334479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.334644 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.334707 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.335571 (+ 864us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.335594 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.335672 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.335832 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260620 07:04:15.340289 19601 raft_consensus.cc:1261] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Refusing update from remote peer cda7b4aa8cc542f8a0a3eae86794b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.340564 19463 raft_consensus.cc:1261] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Refusing update from remote peer cda7b4aa8cc542f8a0a3eae86794b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.341053 19683 consensus_queue.cc:1035] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.341729 19683 consensus_queue.cc:1035] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.354861 19688 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: TxnStatusTablet state changed. Reason: New leader cda7b4aa8cc542f8a0a3eae86794b956. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.355157 19688 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.355604 19685 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: TxnStatusTablet state changed. Reason: New leader cda7b4aa8cc542f8a0a3eae86794b956. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.355886 19685 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.359469 19720 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.359733 19720 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:04:15.360394 19685 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.360615 19685 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P 3d330a5a1d5746d8b842e3fff5b9aae7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.408663 19683 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.409060 19683 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P cda7b4aa8cc542f8a0a3eae86794b956: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:04:15.415091 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.398850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:53421 (local address 127.11.81.193:41511)
0620 07:04:15.399075 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.399082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.407972 (+ 8890us) server_negotiation.cc:407] Connection header received
0620 07:04:15.408302 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.408307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.408398 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.408458 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.409178 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.410369 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.412947 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.413597 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.413687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.414302 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.414427 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.414488 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.414701 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.414780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:04:15.415170 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.398314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.11.81.194:53421)
0620 07:04:15.407837 (+ 9523us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.407858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.408108 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.408691 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.408696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.408708 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:04:15.408999 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.409011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.410924 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.410929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.412791 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.412807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.413847 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.413853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.414030 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.414106 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.414567 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.414581 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.414645 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.414787 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9323,"thread_start_us":9115,"threads_started":1}
I20260620 07:04:15.420152 19688 tablet_replica.cc:439] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } } }
I20260620 07:04:15.420430 19688 tablet_replica.cc:442] T c140d00e912b4651839ec9d5eb84e776 P a5a7c1fd75dc4a2b8f2e5679a1203238: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:04:15.426788 19601 raft_consensus.cc:1261] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Refusing update from remote peer a5a7c1fd75dc4a2b8f2e5679a1203238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.426784 19324 raft_consensus.cc:1261] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Refusing update from remote peer a5a7c1fd75dc4a2b8f2e5679a1203238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.427385 19684 consensus_queue.cc:1035] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.427644 19688 consensus_queue.cc:1035] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.467954 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.461467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46821 (local address 127.11.81.194:35323)
0620 07:04:15.461644 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.461650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.462334 (+ 684us) server_negotiation.cc:407] Connection header received
0620 07:04:15.462546 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.462551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.462633 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.462722 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.463369 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.464514 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.466109 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.466663 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.466787 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.467368 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.467473 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.467541 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.467678 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.467741 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:04:15.468459 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.461524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35323 (local address 127.11.81.193:46821)
0620 07:04:15.462201 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.462246 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.462396 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.462737 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.462742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.462754 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:04:15.463127 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.463140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.464633 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.464637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.465984 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.465996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.466855 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.466862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.467068 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.467142 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.467565 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.467577 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.467668 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.467819 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":292,"threads_started":1}
I20260620 07:04:15.502480 19166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.502753 19166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.505671 19166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Sending DeleteTablet for 3 replicas of tablet 366140897a164e76a1b979fd441b9a85
I20260620 07:04:15.506361 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 366140897a164e76a1b979fd441b9a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260620 07:04:15.506524 19443 tablet_service.cc:1514] Processing DeleteTablet for tablet 366140897a164e76a1b979fd441b9a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:48016
I20260620 07:04:15.506549 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 366140897a164e76a1b979fd441b9a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51792
I20260620 07:04:15.522235 19741 tablet_replica.cc:330] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: stopping tablet replica
I20260620 07:04:15.522487 19741 raft_consensus.cc:2229] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.522797 19741 raft_consensus.cc:2258] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.524122 19741 ts_tablet_manager.cc:1905] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.531864 19741 ts_tablet_manager.cc:1918] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:15.532016 19741 log.cc:1196] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/366140897a164e76a1b979fd441b9a85
I20260620 07:04:15.532473 19741 ts_tablet_manager.cc:1939] T 366140897a164e76a1b979fd441b9a85 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting consensus metadata
I20260620 07:04:15.533674 19152 catalog_manager.cc:4841] TS cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511): tablet 366140897a164e76a1b979fd441b9a85 (table test-table [id=14cf2f6107644ee78c176a9fa5a7834e]) successfully deleted
I20260620 07:04:15.543764 19743 tablet_replica.cc:330] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: stopping tablet replica
I20260620 07:04:15.543982 19743 raft_consensus.cc:2229] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:15.544466 19743 raft_consensus.cc:2258] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.550226 19742 tablet_replica.cc:330] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: stopping tablet replica
I20260620 07:04:15.550436 19742 raft_consensus.cc:2229] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.550673 19742 raft_consensus.cc:2258] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.552299 19742 ts_tablet_manager.cc:1905] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.559563 19742 ts_tablet_manager.cc:1918] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:15.559873 19742 log.cc:1196] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/366140897a164e76a1b979fd441b9a85
I20260620 07:04:15.560037 19743 ts_tablet_manager.cc:1905] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.560456 19742 ts_tablet_manager.cc:1939] T 366140897a164e76a1b979fd441b9a85 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting consensus metadata
I20260620 07:04:15.561581 19151 catalog_manager.cc:4841] TS 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221): tablet 366140897a164e76a1b979fd441b9a85 (table test-table [id=14cf2f6107644ee78c176a9fa5a7834e]) successfully deleted
I20260620 07:04:15.565407 19743 ts_tablet_manager.cc:1918] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:15.565524 19743 log.cc:1196] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/366140897a164e76a1b979fd441b9a85
I20260620 07:04:15.565866 19743 ts_tablet_manager.cc:1939] T 366140897a164e76a1b979fd441b9a85 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting consensus metadata
I20260620 07:04:15.566865 19152 catalog_manager.cc:4841] TS a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323): tablet 366140897a164e76a1b979fd441b9a85 (table test-table [id=14cf2f6107644ee78c176a9fa5a7834e]) successfully deleted
Jun 20 07:04:15 dist-test-slave-wvz3 krb5kdc[19122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:04:15 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:15 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:04:15.620504 19714 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:15.605219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.11.81.254:36895)
0620 07:04:15.605447 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.605454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.605781 (+ 327us) server_negotiation.cc:407] Connection header received
0620 07:04:15.605951 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.605954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.606033 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.606116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:15.607720 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.608433 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.610193 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.610609 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.610699 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.620265 (+ 9566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34104: BlockingRecv error: recv got EOF from 127.0.0.1:34104 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 07:04:15 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM
I20260620 07:04:15.637619 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.623267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.11.81.254:36895)
0620 07:04:15.623478 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.623484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.623760 (+ 276us) server_negotiation.cc:407] Connection header received
0620 07:04:15.623914 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.623919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.624013 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.624132 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:15.625217 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.626277 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.627524 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.628130 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.628259 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.632152 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:15.632157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:15.632160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:15.632196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:15.634686 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:15.635551 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:15.635554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:15.635557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:15.635680 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:15.636071 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:15.636075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:15.636077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:15.636371 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:15.636602 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.636958 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.637232 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 07:04:15.641139 19166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:15.641719 19166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:15.648435 19304 tablet_service.cc:1467] Processing CreateTablet for tablet eea60c993e0540a3b773df3000312ee9 (DEFAULT_TABLE table=test-table [id=7af2506af2cd4b1caf186bb3e08972dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.648510 19581 tablet_service.cc:1467] Processing CreateTablet for tablet eea60c993e0540a3b773df3000312ee9 (DEFAULT_TABLE table=test-table [id=7af2506af2cd4b1caf186bb3e08972dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.648921 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea60c993e0540a3b773df3000312ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.648959 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea60c993e0540a3b773df3000312ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.653275 19679 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Bootstrap starting.
I20260620 07:04:15.654466 19679 tablet_bootstrap.cc:654] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.655717 19443 tablet_service.cc:1467] Processing CreateTablet for tablet eea60c993e0540a3b773df3000312ee9 (DEFAULT_TABLE table=test-table [id=7af2506af2cd4b1caf186bb3e08972dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.656163 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea60c993e0540a3b773df3000312ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.656694 19679 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: No bootstrap required, opened a new log
I20260620 07:04:15.656829 19679 ts_tablet_manager.cc:1397] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 07:04:15.657276 19678 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Bootstrap starting.
I20260620 07:04:15.657562 19679 raft_consensus.cc:348] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.657796 19679 raft_consensus.cc:374] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.657881 19679 raft_consensus.cc:729] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.658092 19679 consensus_queue.cc:260] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.659008 19678 tablet_bootstrap.cc:654] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.659318 19679 ts_tablet_manager.cc:1428] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:04:15.664670 19677 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Bootstrap starting.
I20260620 07:04:15.666260 19677 tablet_bootstrap.cc:654] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.668454 19677 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: No bootstrap required, opened a new log
I20260620 07:04:15.668602 19677 ts_tablet_manager.cc:1397] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:04:15.669359 19677 raft_consensus.cc:348] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.669615 19677 raft_consensus.cc:374] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.669714 19677 raft_consensus.cc:729] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.669999 19677 consensus_queue.cc:260] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.670594 19677 ts_tablet_manager.cc:1428] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:15.673012 19678 tablet_bootstrap.cc:492] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: No bootstrap required, opened a new log
I20260620 07:04:15.673183 19678 ts_tablet_manager.cc:1397] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260620 07:04:15.673969 19678 raft_consensus.cc:348] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.674206 19678 raft_consensus.cc:374] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.674305 19678 raft_consensus.cc:729] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.674527 19678 consensus_queue.cc:260] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.675483 19678 ts_tablet_manager.cc:1428] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:04:15.685993 19685 raft_consensus.cc:482] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:15.686251 19685 raft_consensus.cc:504] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.688361 19685 leader_election.cc:288] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323)
I20260620 07:04:15.696729 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.689503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:54595 (local address 127.11.81.194:35323)
0620 07:04:15.689685 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.689692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.691483 (+ 1791us) server_negotiation.cc:407] Connection header received
0620 07:04:15.691639 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.691644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.691718 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.691809 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:15.692469 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.693324 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.694997 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.695828 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.695987 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.696063 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.696327 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.696456 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:04:15.696833 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.689351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35323 (local address 127.11.81.195:54595)
0620 07:04:15.691381 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.691396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.691506 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.691963 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.691968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.691991 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:15.692323 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.692332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.693450 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.693455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.694838 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.694851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.696029 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.696035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.696176 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.696199 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.696290 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.696424 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"mutex_wait_us":3,"thread_start_us":1711,"threads_started":1}
I20260620 07:04:15.697727 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.688622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:60277 (local address 127.11.81.193:41511)
0620 07:04:15.688864 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.688870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.689288 (+ 418us) server_negotiation.cc:407] Connection header received
0620 07:04:15.689596 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.689607 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.689691 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.689778 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:15.690656 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.691902 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.695080 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.696460 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.696661 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.696732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.697303 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.697452 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:04:15.697829 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.688641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.11.81.195:60277)
0620 07:04:15.689199 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.689217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.689407 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.689987 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.689993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.690022 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:15.690469 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.690481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.692115 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.692122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.694865 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.694882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.696898 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.696905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.697104 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.697139 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.697258 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.697395 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":427,"thread_start_us":293,"threads_started":1}
I20260620 07:04:15.697911 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea60c993e0540a3b773df3000312ee9" candidate_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" is_pre_election: true
I20260620 07:04:15.698148 19463 raft_consensus.cc:2454] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d330a5a1d5746d8b842e3fff5b9aae7 in term 0.
I20260620 07:04:15.698706 19534 leader_election.cc:302] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d330a5a1d5746d8b842e3fff5b9aae7, a5a7c1fd75dc4a2b8f2e5679a1203238; no voters:
I20260620 07:04:15.698835 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea60c993e0540a3b773df3000312ee9" candidate_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" is_pre_election: true
I20260620 07:04:15.699064 19324 raft_consensus.cc:2454] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d330a5a1d5746d8b842e3fff5b9aae7 in term 0.
I20260620 07:04:15.699177 19685 raft_consensus.cc:2781] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:15.699316 19685 raft_consensus.cc:482] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:15.699414 19685 raft_consensus.cc:3037] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.701918 19685 raft_consensus.cc:504] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.702487 19685 leader_election.cc:288] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [CANDIDATE]: Term 1 election: Requested vote from peers cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511), a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323)
I20260620 07:04:15.702934 19324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea60c993e0540a3b773df3000312ee9" candidate_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda7b4aa8cc542f8a0a3eae86794b956"
I20260620 07:04:15.703130 19324 raft_consensus.cc:3037] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.703052 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea60c993e0540a3b773df3000312ee9" candidate_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
I20260620 07:04:15.703265 19463 raft_consensus.cc:3037] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.705271 19324 raft_consensus.cc:2454] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d330a5a1d5746d8b842e3fff5b9aae7 in term 1.
I20260620 07:04:15.705621 19534 leader_election.cc:302] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d330a5a1d5746d8b842e3fff5b9aae7, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.705863 19685 raft_consensus.cc:2781] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:15.706146 19463 raft_consensus.cc:2454] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d330a5a1d5746d8b842e3fff5b9aae7 in term 1.
I20260620 07:04:15.706367 19685 raft_consensus.cc:686] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 LEADER]: Becoming Leader. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Running, Role: LEADER
I20260620 07:04:15.706790 19685 consensus_queue.cc:237] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.710474 19167 catalog_manager.cc:5495] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } health_report { overall_health: HEALTHY } } }
I20260620 07:04:15.737709 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.730830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.11.81.195:39221)
0620 07:04:15.731006 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.731010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.731104 (+ 94us) server_negotiation.cc:407] Connection header received
0620 07:04:15.731418 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.731422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.731514 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.731641 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.732077 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.732789 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.734641 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.735210 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.735384 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.735997 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.736752 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.736901 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.737324 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.737410 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:04:15.741262 19324 raft_consensus.cc:1261] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Refusing update from remote peer 3d330a5a1d5746d8b842e3fff5b9aae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.741390 19463 raft_consensus.cc:1261] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Refusing update from remote peer 3d330a5a1d5746d8b842e3fff5b9aae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.742022 19761 consensus_queue.cc:1035] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.742479 19685 consensus_queue.cc:1035] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.760042 19167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.760406 19167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.764456 19167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Sending DeleteTablet for 3 replicas of tablet eea60c993e0540a3b773df3000312ee9
I20260620 07:04:15.765489 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet eea60c993e0540a3b773df3000312ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260620 07:04:15.765897 19741 tablet_replica.cc:330] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: stopping tablet replica
I20260620 07:04:15.766109 19741 raft_consensus.cc:2229] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.766254 19741 pending_rounds.cc:62] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Trying to abort 2 pending ops.
I20260620 07:04:15.766444 19443 tablet_service.cc:1514] Processing DeleteTablet for tablet eea60c993e0540a3b773df3000312ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:48016
I20260620 07:04:15.766350 19741 pending_rounds.cc:68] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298822372309483520 op_type: WRITE_OP write_request { tablet_id: "eea60c993e0540a3b773df3000312ee9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\233\371\330\321\006\"8\n\ntest-admin\022*\n 7af2506af2cd4b1caf186bb3e08972dc\020\001\030\001 \001(\001" signature: "<redacted>""9\341,\253t\375\025#H\220tjk\324X\221\211\235\tI\243\034\007P\335\275\221\311\t\262\240{&\004>v\035\326\334WQ5\240\007h`\342\200\320R\026q\254\004\305\206\256\267\230\317\022\337g\312" signing_key_seq_num: 0 } } request_id { client_id: "273ffc6b9661401f9c0cbb43d6de5961" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 07:04:15.766723 19743 tablet_replica.cc:330] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: stopping tablet replica
I20260620 07:04:15.766902 19743 raft_consensus.cc:2229] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.766997 19741 pending_rounds.cc:68] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298822372178907136 op_type: NO_OP noop_request { }
I20260620 07:04:15.767115 19743 raft_consensus.cc:2258] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.767158 19741 raft_consensus.cc:2866] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 07:04:15.767266 19741 raft_consensus.cc:2258] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.768594 19743 ts_tablet_manager.cc:1905] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.773077 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet eea60c993e0540a3b773df3000312ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51792
I20260620 07:04:15.773357 19742 tablet_replica.cc:330] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: stopping tablet replica
I20260620 07:04:15.773568 19742 raft_consensus.cc:2229] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 LEADER]: Raft consensus shutting down.
W20260620 07:04:15.773620 19534 consensus_peers.cc:487] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 -> Peer cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511): Couldn't send request to peer cda7b4aa8cc542f8a0a3eae86794b956. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:04:15.774312 19742 raft_consensus.cc:2258] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.774489 19741 ts_tablet_manager.cc:1905] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.775558 19742 ts_tablet_manager.cc:1905] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:15.781311 19741 ts_tablet_manager.cc:1918] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:15.781463 19741 log.cc:1196] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/eea60c993e0540a3b773df3000312ee9
I20260620 07:04:15.781880 19741 ts_tablet_manager.cc:1939] T eea60c993e0540a3b773df3000312ee9 P cda7b4aa8cc542f8a0a3eae86794b956: Deleting consensus metadata
I20260620 07:04:15.782783 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet eea60c993e0540a3b773df3000312ee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51792
I20260620 07:04:15.783043 19152 catalog_manager.cc:4841] TS cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511): tablet eea60c993e0540a3b773df3000312ee9 (table test-table [id=7af2506af2cd4b1caf186bb3e08972dc]) successfully deleted
I20260620 07:04:15.784845 19742 ts_tablet_manager.cc:1918] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:15.784991 19742 log.cc:1196] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/eea60c993e0540a3b773df3000312ee9
I20260620 07:04:15.785388 19742 ts_tablet_manager.cc:1939] T eea60c993e0540a3b773df3000312ee9 P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting consensus metadata
I20260620 07:04:15.786645 19166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:15.787364 19166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:15.787647 19743 ts_tablet_manager.cc:1918] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:15.787793 19743 log.cc:1196] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/eea60c993e0540a3b773df3000312ee9
I20260620 07:04:15.788164 19743 ts_tablet_manager.cc:1939] T eea60c993e0540a3b773df3000312ee9 P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting consensus metadata
I20260620 07:04:15.788635 19151 catalog_manager.cc:4841] TS 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221): tablet eea60c993e0540a3b773df3000312ee9 (table test-table [id=7af2506af2cd4b1caf186bb3e08972dc]) successfully deleted
W20260620 07:04:15.789084 19151 catalog_manager.cc:4808] TS 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221): delete failed for tablet eea60c993e0540a3b773df3000312ee9 because the tablet was not found. No further retry: Not found: Tablet not found: eea60c993e0540a3b773df3000312ee9
I20260620 07:04:15.789250 19152 catalog_manager.cc:4841] TS a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323): tablet eea60c993e0540a3b773df3000312ee9 (table test-table [id=7af2506af2cd4b1caf186bb3e08972dc]) successfully deleted
I20260620 07:04:15.796243 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 7843801e52324d84ba2d51ead89638de (DEFAULT_TABLE table=test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.796782 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7843801e52324d84ba2d51ead89638de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.802987 19443 tablet_service.cc:1467] Processing CreateTablet for tablet 7843801e52324d84ba2d51ead89638de (DEFAULT_TABLE table=test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.803473 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7843801e52324d84ba2d51ead89638de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.803575 19678 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Bootstrap starting.
I20260620 07:04:15.805253 19678 tablet_bootstrap.cc:654] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.807749 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 7843801e52324d84ba2d51ead89638de (DEFAULT_TABLE table=test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:15.808182 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7843801e52324d84ba2d51ead89638de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:15.815158 19679 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Bootstrap starting.
I20260620 07:04:15.816931 19679 tablet_bootstrap.cc:654] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.818109 19677 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Bootstrap starting.
I20260620 07:04:15.819706 19677 tablet_bootstrap.cc:654] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:15.821907 19677 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: No bootstrap required, opened a new log
I20260620 07:04:15.822064 19677 ts_tablet_manager.cc:1397] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:04:15.822340 19679 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: No bootstrap required, opened a new log
I20260620 07:04:15.822517 19679 ts_tablet_manager.cc:1397] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260620 07:04:15.822842 19677 raft_consensus.cc:348] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.823102 19677 raft_consensus.cc:374] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.823184 19677 raft_consensus.cc:729] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a7c1fd75dc4a2b8f2e5679a1203238, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.823346 19678 tablet_bootstrap.cc:492] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: No bootstrap required, opened a new log
I20260620 07:04:15.823563 19678 ts_tablet_manager.cc:1397] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.009s
I20260620 07:04:15.823486 19677 consensus_queue.cc:260] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.824317 19679 raft_consensus.cc:348] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.824409 19678 raft_consensus.cc:348] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.824589 19679 raft_consensus.cc:374] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.824676 19679 raft_consensus.cc:729] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d330a5a1d5746d8b842e3fff5b9aae7, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.824780 19678 raft_consensus.cc:374] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:15.824891 19678 raft_consensus.cc:729] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Initialized, Role: FOLLOWER
I20260620 07:04:15.824955 19679 consensus_queue.cc:260] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.825217 19678 consensus_queue.cc:260] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.825764 19678 ts_tablet_manager.cc:1428] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:04:15.826288 19679 ts_tablet_manager.cc:1428] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:04:15.827405 19677 ts_tablet_manager.cc:1428] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 07:04:15.865352 19727 raft_consensus.cc:482] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:15.865619 19727 raft_consensus.cc:504] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.866386 19727 leader_election.cc:288] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.866938 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7843801e52324d84ba2d51ead89638de" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" is_pre_election: true
I20260620 07:04:15.866972 19601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7843801e52324d84ba2d51ead89638de" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" is_pre_election: true
I20260620 07:04:15.867182 19463 raft_consensus.cc:2454] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 0.
I20260620 07:04:15.867306 19601 raft_consensus.cc:2454] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 0.
I20260620 07:04:15.867614 19255 leader_election.cc:302] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.867888 19727 raft_consensus.cc:2781] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:15.868002 19727 raft_consensus.cc:482] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:15.868088 19727 raft_consensus.cc:3037] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.870528 19727 raft_consensus.cc:504] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.871169 19727 leader_election.cc:288] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 election: Requested vote from peers a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323), 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221)
I20260620 07:04:15.871762 19601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7843801e52324d84ba2d51ead89638de" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7"
I20260620 07:04:15.871886 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7843801e52324d84ba2d51ead89638de" candidate_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238"
I20260620 07:04:15.872023 19601 raft_consensus.cc:3037] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.872090 19463 raft_consensus.cc:3037] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:15.874063 19463 raft_consensus.cc:2454] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 1.
I20260620 07:04:15.874440 19255 leader_election.cc:302] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a7c1fd75dc4a2b8f2e5679a1203238, cda7b4aa8cc542f8a0a3eae86794b956; no voters:
I20260620 07:04:15.874716 19727 raft_consensus.cc:2781] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:15.874855 19727 raft_consensus.cc:686] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 LEADER]: Becoming Leader. State: Replica: cda7b4aa8cc542f8a0a3eae86794b956, State: Running, Role: LEADER
I20260620 07:04:15.875098 19601 raft_consensus.cc:2454] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda7b4aa8cc542f8a0a3eae86794b956 in term 1.
I20260620 07:04:15.875123 19727 consensus_queue.cc:237] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } }
I20260620 07:04:15.877807 19164 catalog_manager.cc:5495] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193). New cstate: current_term: 1 leader_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda7b4aa8cc542f8a0a3eae86794b956" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 } health_report { overall_health: UNKNOWN } } }
I20260620 07:04:15.900352 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.895387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.11.81.193:41511)
0620 07:04:15.895592 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.895599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.895771 (+ 172us) server_negotiation.cc:407] Connection header received
0620 07:04:15.895951 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.895957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.896037 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.896122 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.896499 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.897269 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.898630 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.899116 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.899232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.899687 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.899802 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.899874 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.900012 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.900088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:04:15.914139 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.908299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:48445 (local address 127.11.81.193:41511)
0620 07:04:15.908691 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.908697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:15.908717 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:04:15.908773 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:15.908777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:15.908853 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:15.908942 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:15.909283 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.910420 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.912228 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.912890 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.913010 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.913405 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.913536 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.913636 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:15.913746 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 07:04:15.913828 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260620 07:04:15.914157 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.908226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:41511 (local address 127.11.81.193:48445)
0620 07:04:15.908500 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:15.908521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:15.908631 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:15.908959 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:15.908964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:15.908976 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:04:15.909158 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.909167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.910539 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.910545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.912078 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:15.912092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:15.913030 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:15.913036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:15.913197 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:15.913244 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:04:15.913633 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:04:15.913643 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:04:15.913720 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:04:15.913864 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 07:04:15.924561 19463 raft_consensus.cc:1261] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Refusing update from remote peer cda7b4aa8cc542f8a0a3eae86794b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.924767 19601 raft_consensus.cc:1261] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Refusing update from remote peer cda7b4aa8cc542f8a0a3eae86794b956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:15.925067 19720 consensus_queue.cc:1035] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a7c1fd75dc4a2b8f2e5679a1203238" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.925791 19720 consensus_queue.cc:1035] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d330a5a1d5746d8b842e3fff5b9aae7" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:15.985806 19164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.986104 19164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:15.990057 19164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e70eb8639c6249998b3b6e3e53fcb572: Sending DeleteTablet for 3 replicas of tablet 7843801e52324d84ba2d51ead89638de
I20260620 07:04:15.991184 19443 tablet_service.cc:1514] Processing DeleteTablet for tablet 7843801e52324d84ba2d51ead89638de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:48016
I20260620 07:04:15.991940 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 7843801e52324d84ba2d51ead89638de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51792
I20260620 07:04:15.992254 19742 tablet_replica.cc:330] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: stopping tablet replica
I20260620 07:04:15.992285 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19238
I20260620 07:04:15.992445 19742 raft_consensus.cc:2229] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.992673 19742 raft_consensus.cc:2258] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.993497 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 7843801e52324d84ba2d51ead89638de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:15 UTC) from {username='oryx', principal='oryx/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260620 07:04:15.993825 19741 tablet_replica.cc:330] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: stopping tablet replica
I20260620 07:04:15.994025 19741 raft_consensus.cc:2229] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:15.994571 19741 raft_consensus.cc:2258] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.995038 19743 tablet_replica.cc:330] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: stopping tablet replica
I20260620 07:04:15.995572 19743 raft_consensus.cc:2229] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:15.995824 19743 raft_consensus.cc:2258] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:15.997191 19743 ts_tablet_manager.cc:1905] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:16.004047 19741 ts_tablet_manager.cc:1905] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:16.008495 19742 ts_tablet_manager.cc:1905] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.193@KRBTEST.COM
I20260620 07:04:16.019887 19743 ts_tablet_manager.cc:1918] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:16.020776 19742 ts_tablet_manager.cc:1918] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:16.022130 19741 ts_tablet_manager.cc:1918] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:04:16.022375 19743 log.cc:1196] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/7843801e52324d84ba2d51ead89638de
I20260620 07:04:16.022670 19742 log.cc:1196] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/7843801e52324d84ba2d51ead89638de
I20260620 07:04:16.023203 19743 ts_tablet_manager.cc:1939] T 7843801e52324d84ba2d51ead89638de P a5a7c1fd75dc4a2b8f2e5679a1203238: Deleting consensus metadata
I20260620 07:04:16.023774 19742 ts_tablet_manager.cc:1939] T 7843801e52324d84ba2d51ead89638de P 3d330a5a1d5746d8b842e3fff5b9aae7: Deleting consensus metadata
I20260620 07:04:16.025244 19152 catalog_manager.cc:4841] TS a5a7c1fd75dc4a2b8f2e5679a1203238 (127.11.81.194:35323): tablet 7843801e52324d84ba2d51ead89638de (table test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]) successfully deleted
I20260620 07:04:16.025681 19741 log.cc:1196] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/7843801e52324d84ba2d51ead89638de
I20260620 07:04:16.027040 19151 catalog_manager.cc:4841] TS 3d330a5a1d5746d8b842e3fff5b9aae7 (127.11.81.195:39221): tablet 7843801e52324d84ba2d51ead89638de (table test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]) successfully deleted
I20260620 07:04:16.027750 19741 ts_tablet_manager.cc:1939] T 7843801e52324d84ba2d51ead89638de P cda7b4aa8cc542f8a0a3eae86794b956: Deleting consensus metadata
I20260620 07:04:16.028165 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:15.994397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.11.81.193:41511)
0620 07:04:15.994600 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:15.994606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:16.008766 (+ 14160us) server_negotiation.cc:407] Connection header received
0620 07:04:16.008983 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:16.008989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:16.009084 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:16.009157 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:16.011003 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.012246 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.014018 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.014708 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.014810 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:16.020292 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:16.020311 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:16.020332 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:16.020378 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:16.024432 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.025414 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.025424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.025427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.025627 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.026109 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.026114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.026118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.026619 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:16.026944 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:16.027555 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 07:04:16.027808 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:04:16.029610 19152 catalog_manager.cc:4841] TS cda7b4aa8cc542f8a0a3eae86794b956 (127.11.81.193:41511): tablet 7843801e52324d84ba2d51ead89638de (table test-table [id=e72eb97a4b2f4b459a2854c77daab0e0]) successfully deleted
I20260620 07:04:16.029966 19369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:16.180291 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19238
I20260620 07:04:16.199057 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19378
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.194@KRBTEST.COM
I20260620 07:04:16.216357 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:16.199878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.11.81.194:35323)
0620 07:04:16.200335 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:16.200341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:16.200361 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:04:16.200425 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:16.200430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:16.200508 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:16.200637 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:16.201794 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.202894 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.204467 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.205114 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.205242 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:16.209406 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:16.209424 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:16.209445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:16.209498 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:16.213252 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.214042 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.214053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.214057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.214221 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.214618 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.214623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.214627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.215078 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:16.215402 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:16.215746 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:04:16.215977 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260620 07:04:16.217082 19508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:16.340502 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19378
I20260620 07:04:16.358482 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19517
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.81.195@KRBTEST.COM
I20260620 07:04:16.372735 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:16.359160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.11.81.195:39221)
0620 07:04:16.359598 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:16.359602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:16.359619 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:04:16.359676 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:16.359679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:16.359738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:16.359836 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:16.361055 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.361803 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.363023 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:16.363553 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:16.363657 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:16.367789 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:16.367804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:16.367821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:16.367855 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:16.370433 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.371023 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.371030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.371033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.371174 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:16.371534 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:16.371537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:16.371540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:16.371884 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:16.372116 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:16.372317 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 07:04:16.372482 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260620 07:04:16.373361 19647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:16.496497 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19517
I20260620 07:04:16.513844 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19135
I20260620 07:04:16.514984 19207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:16.627980 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19135
2026-06-20T07:04:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6387 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19784](info): set up 2 sockets
Jun 20 07:04:16 dist-test-slave-wvz3 krb5kdc[19784](info): commencing operation
krb5kdc: starting...
W20260620 07:04:17.643962 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:17 dist-test-slave-wvz3 krb5kdc[19784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:04:17.867870 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36939
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:36939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:18.187384 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:18.187686 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:18.187821 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:18.195165 19797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:18.195309 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:18.195385 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:18.195474 19797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:18.195536 19797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:18.205701 19797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:36939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:18.207060 19797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:18.208451 19797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:18.220021 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:18.219928 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:18.219959 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:18.220894 19797 server_base.cc:1034] running on GCE node
I20260620 07:04:18.221689 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:18.223163 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:18.224382 19797 hybrid_clock.cc:648] HybridClock initialized: now 1781939058224337 us; error 63 us; skew 500 ppm
I20260620 07:04:18.224743 19797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T07:04:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1660 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:18 dist-test-slave-wvz3 krb5kdc[19811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:18 dist-test-slave-wvz3 krb5kdc[19811](info): set up 2 sockets
Jun 20 07:04:18 dist-test-slave-wvz3 krb5kdc[19811](info): commencing operation
krb5kdc: starting...
W20260620 07:04:19.311996 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.009s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:19 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:04:19.524837 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43279
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:19.843886 19824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:19.844192 19824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:19.844287 19824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:19.851711 19824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:19.851820 19824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:19.851871 19824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:19.851926 19824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:19.851975 19824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:19.861811 19824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:19.863099 19824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:19.864374 19824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:19.875684 19829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:19.875751 19830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:19.875725 19832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:19.876417 19824 server_base.cc:1034] running on GCE node
I20260620 07:04:19.877175 19824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:19.878618 19824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:19.879817 19824 hybrid_clock.cc:648] HybridClock initialized: now 1781939059879749 us; error 60 us; skew 500 ppm
I20260620 07:04:19.880168 19824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:19 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:19.883113 19824 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:04:19.884620 19824 webserver.cc:460] Webserver started at http://127.11.81.254:34883/ using document root <none> and password file <none>
I20260620 07:04:19.885371 19824 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:19.885510 19824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:19.885885 19824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:19.888890 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "475b732a52494932a2af6cdc40113e64"
format_stamp: "Formatted at 2026-06-20 07:04:19 on dist-test-slave-wvz3"
server_key: "3a54a5c3e94517163eb26eac87fd96d0"
server_key_iv: "3d00365fca3c3b650eddb9ece77b894a"
server_key_version: "encryptionkey@0"
I20260620 07:04:19.889573 19824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "475b732a52494932a2af6cdc40113e64"
format_stamp: "Formatted at 2026-06-20 07:04:19 on dist-test-slave-wvz3"
server_key: "3a54a5c3e94517163eb26eac87fd96d0"
server_key_iv: "3d00365fca3c3b650eddb9ece77b894a"
server_key_version: "encryptionkey@0"
I20260620 07:04:19.894650 19824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:04:19.898078 19839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:19.899425 19824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:04:19.899593 19824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "475b732a52494932a2af6cdc40113e64"
format_stamp: "Formatted at 2026-06-20 07:04:19 on dist-test-slave-wvz3"
server_key: "3a54a5c3e94517163eb26eac87fd96d0"
server_key_iv: "3d00365fca3c3b650eddb9ece77b894a"
server_key_version: "encryptionkey@0"
I20260620 07:04:19.899780 19824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:19.942845 19824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:19.947250 19824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:19.947552 19824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:19.964309 19824 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43279
I20260620 07:04:19.964385 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43279 every 8 connection(s)
I20260620 07:04:19.965700 19824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:19.969556 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:19.974855 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19824
I20260620 07:04:19.975070 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:19.975868 11591 external_mini_cluster.cc:1428] Setting key 107e8fe9c36f3d3c14984486add7bcfa
I20260620 07:04:19.981118 19891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Bootstrap starting.
I20260620 07:04:19.984213 19891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:19.985220 19891 log.cc:824] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:19.988262 19891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: No bootstrap required, opened a new log
Jun 20 07:04:19 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:19.998153 19891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:19.998656 19891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:19.998770 19891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475b732a52494932a2af6cdc40113e64, State: Initialized, Role: FOLLOWER
I20260620 07:04:19.999522 19891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:19.999833 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:19.978026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.11.81.254:43279)
0620 07:04:19.978820 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:19.978833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:19.978873 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:04:19.980744 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:19.980772 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:19.981359 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:19.981745 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:19.983264 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:19.984641 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:19.985986 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:19.986842 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:19.987017 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:19.991141 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:19.991153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:19.991174 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:19.991229 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:19.995125 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:19.995872 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:19.995882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:19.995893 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:19.996137 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:19.996504 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:19.996509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:19.996512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:19.997292 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:19.997580 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:19.997923 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:04:19.998395 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":342,"threads_started":1}
I20260620 07:04:19.999830 19891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:20.000013 19891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:20.000200 19891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:20.002847 19891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:20.003479 19891 leader_election.cc:302] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475b732a52494932a2af6cdc40113e64; no voters:
I20260620 07:04:20.004123 19891 leader_election.cc:288] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:20.004259 19896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:20.004938 19896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 LEADER]: Becoming Leader. State: Replica: 475b732a52494932a2af6cdc40113e64, State: Running, Role: LEADER
I20260620 07:04:20.005466 19896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:20.006285 19891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:20.008327 19898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475b732a52494932a2af6cdc40113e64. Latest consensus state: current_term: 1 leader_uuid: "475b732a52494932a2af6cdc40113e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } } }
I20260620 07:04:20.008765 19898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:20.010874 19903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:20.010650 19897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "475b732a52494932a2af6cdc40113e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } } }
I20260620 07:04:20.011023 19897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:20.016390 19903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:20.027089 19903 catalog_manager.cc:1347] Generated new cluster ID: 57f9398a57c94c0c95968b8704042eff
I20260620 07:04:20.027222 19903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:20.037063 19903 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:20.038407 19903 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:20.050419 19903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Generated new TSK 0
I20260620 07:04:20.051682 19903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:04:20.110836 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:20.437664 19917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:20.437954 19917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:20.438048 19917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:20.445345 19917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:20.445468 19917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:20.445595 19917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:20.455438 19917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:20.456801 19917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:20.458043 19917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:20.470782 19922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:20.470799 19923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:20.470963 19925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:20.472549 19917 server_base.cc:1034] running on GCE node
I20260620 07:04:20.473176 19917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:20.474149 19917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:20.475379 19917 hybrid_clock.cc:648] HybridClock initialized: now 1781939060475310 us; error 58 us; skew 500 ppm
I20260620 07:04:20.475694 19917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:20 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939060, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:20.478960 19917 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:04:20.480446 19917 webserver.cc:460] Webserver started at http://127.11.81.193:35363/ using document root <none> and password file <none>
I20260620 07:04:20.481177 19917 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:20.481302 19917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:20.481606 19917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:20.484683 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "7443562fa6cf4a7c9f27d95c9e23bb48"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "5a4eba552ebc071da612ccccb7428ecc"
server_key_iv: "cd8d66895e63dda216b2772f5c889c39"
server_key_version: "encryptionkey@0"
I20260620 07:04:20.485513 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "7443562fa6cf4a7c9f27d95c9e23bb48"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "5a4eba552ebc071da612ccccb7428ecc"
server_key_iv: "cd8d66895e63dda216b2772f5c889c39"
server_key_version: "encryptionkey@0"
I20260620 07:04:20.491158 19917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:04:20.494928 19932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:20.496449 19917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:20.496623 19917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "7443562fa6cf4a7c9f27d95c9e23bb48"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "5a4eba552ebc071da612ccccb7428ecc"
server_key_iv: "cd8d66895e63dda216b2772f5c889c39"
server_key_version: "encryptionkey@0"
I20260620 07:04:20.496825 19917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:20.529227 19917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:20.533371 19917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:20.533643 19917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:20.534636 19917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:20.536039 19917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:20.536125 19917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:20.536247 19917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:20.536298 19917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:20.566499 19917 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:36241
I20260620 07:04:20.566530 20044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:36241 every 8 connection(s)
I20260620 07:04:20.567842 19917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:20.572777 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 19917
I20260620 07:04:20.572995 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:20.573601 11591 external_mini_cluster.cc:1428] Setting key 7064907f04962d378c38e6e69d68a4e6
Jun 20 07:04:20 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939060, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:20.588856 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:20.570405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46841 (local address 127.11.81.254:43279)
0620 07:04:20.570845 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:20.570852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:20.571836 (+ 984us) server_negotiation.cc:407] Connection header received
0620 07:04:20.573725 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:20.573731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:20.573817 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:20.573944 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:20.575767 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:20.576746 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:20.578356 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:20.578879 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:20.578993 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:20.582888 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:20.582893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:20.582897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:20.582935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:20.585525 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:20.586323 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:20.586327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:20.586330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:20.586456 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:20.586903 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:20.586906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:20.586908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:20.587182 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:20.587360 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:20.588236 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 07:04:20.588477 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":197,"threads_started":1}
I20260620 07:04:20.589879 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:20.570966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:46841)
0620 07:04:20.571688 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:20.571726 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:20.573447 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:20.574195 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:20.574203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:20.574893 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:20.575603 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:20.575625 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:20.576898 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:20.576902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:20.578224 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:20.578236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:20.579079 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:20.579084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:20.579325 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:20.580048 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:20.580084 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:20.582603 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:20.585693 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:20.585703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:20.585723 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:20.586177 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:20.586570 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:20.586575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:20.586579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:20.586782 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:20.587360 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:20.587369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:20.587842 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 07:04:20.588099 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 07:04:20.588684 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":300,"threads_started":1}
I20260620 07:04:20.591641 20045 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43279
I20260620 07:04:20.591967 20045 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:20.592696 20045 heartbeater.cc:511] Master 127.11.81.254:43279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:04:20.594959 19856 ts_manager.cc:194] Registered new tserver with Master: 7443562fa6cf4a7c9f27d95c9e23bb48 (127.11.81.193:36241)
I20260620 07:04:20.596487 19856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:46841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:04:20.623332 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:20.943341 20051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:20.943634 20051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:20.943756 20051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:20.950940 20051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:20.951071 20051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:20.951223 20051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:20.961212 20051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:20.962496 20051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:20.963829 20051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:20.979211 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:20.979481 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:20.979424 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:20.981223 20051 server_base.cc:1034] running on GCE node
I20260620 07:04:20.981926 20051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:20.982985 20051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:20.984201 20051 hybrid_clock.cc:648] HybridClock initialized: now 1781939060984144 us; error 48 us; skew 500 ppm
I20260620 07:04:20.984545 20051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:20 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939060, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:20.991199 20051 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:04:20.992837 20051 webserver.cc:460] Webserver started at http://127.11.81.194:39699/ using document root <none> and password file <none>
I20260620 07:04:20.993552 20051 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:20.993671 20051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:20.993990 20051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:20.997144 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "71bc01596add4e62a32ad617eef2ed19"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "b285016e5ae5aa73a2fa9e69d0f92c10"
server_key_iv: "0ea135a57f85df2a03d4aab419d7eb6a"
server_key_version: "encryptionkey@0"
I20260620 07:04:20.997912 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "71bc01596add4e62a32ad617eef2ed19"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "b285016e5ae5aa73a2fa9e69d0f92c10"
server_key_iv: "0ea135a57f85df2a03d4aab419d7eb6a"
server_key_version: "encryptionkey@0"
I20260620 07:04:21.003388 20051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:04:21.007225 20066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.008718 20051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:04:21.008885 20051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "71bc01596add4e62a32ad617eef2ed19"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "b285016e5ae5aa73a2fa9e69d0f92c10"
server_key_iv: "0ea135a57f85df2a03d4aab419d7eb6a"
server_key_version: "encryptionkey@0"
I20260620 07:04:21.009079 20051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:21.030787 20051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:21.034909 20051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:21.035183 20051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:21.036168 20051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:21.037561 20051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:21.037643 20051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.037765 20051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:21.037818 20051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.067003 20051 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:42623
I20260620 07:04:21.067046 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:42623 every 8 connection(s)
I20260620 07:04:21.068311 20051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:21.073693 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20051
I20260620 07:04:21.073832 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:21.074280 11591 external_mini_cluster.cc:1428] Setting key 98af2b4470cf805988d0b443fad3063a
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939060, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:21.087677 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.070931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:48729 (local address 127.11.81.254:43279)
0620 07:04:21.071117 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:21.071121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:21.072246 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 07:04:21.074160 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:21.074166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:21.074243 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:21.074327 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:21.076103 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.076810 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.078310 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.078742 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.078852 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.082086 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:21.082091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:21.082095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:21.082127 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:21.084742 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.085454 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.085458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.085460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.085586 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.085983 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.085986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.085988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.086258 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:21.086390 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:21.087067 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:04:21.087333 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:04:21.088661 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.071393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:48729)
0620 07:04:21.072093 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:21.072129 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:21.073875 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:21.074568 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:21.074576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:21.075154 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:21.075945 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:21.075971 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.076951 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.076957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:21.078170 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:21.078182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.078948 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.078955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:21.079180 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.079830 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:21.079857 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:21.081875 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:21.084884 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:21.084893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:21.084911 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:21.085332 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:21.085687 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:21.085692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:21.085696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:21.085873 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:21.086382 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:21.086390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:21.086722 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 07:04:21.086958 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 07:04:21.087525 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":252,"threads_started":1}
I20260620 07:04:21.090245 20179 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43279
I20260620 07:04:21.090533 20179 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:21.091374 20179 heartbeater.cc:511] Master 127.11.81.254:43279 requested a full tablet report, sending...
I20260620 07:04:21.093029 19856 ts_manager.cc:194] Registered new tserver with Master: 71bc01596add4e62a32ad617eef2ed19 (127.11.81.194:42623)
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:04:21.094144 19856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:48729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:04:21.120671 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:21.439883 20184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:21.440289 20184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:21.440443 20184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:21.449012 20184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:21.449146 20184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:21.449307 20184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:21.459272 20184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:21.460598 20184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:21.461933 20184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:21.474632 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:21.474663 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:21.474725 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:21.475716 20184 server_base.cc:1034] running on GCE node
I20260620 07:04:21.476331 20184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:21.477372 20184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:21.478557 20184 hybrid_clock.cc:648] HybridClock initialized: now 1781939061478505 us; error 49 us; skew 500 ppm
I20260620 07:04:21.478937 20184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939061, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:21.482443 20184 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:04:21.484380 20184 webserver.cc:460] Webserver started at http://127.11.81.195:43609/ using document root <none> and password file <none>
I20260620 07:04:21.485339 20184 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:21.485478 20184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:21.485818 20184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:21.488919 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "4202d538077f4483ad230607c23c9d31"
format_stamp: "Formatted at 2026-06-20 07:04:21 on dist-test-slave-wvz3"
server_key: "d625a53f04339cb92f2b6df42e80b4b9"
server_key_iv: "82cd4aca18d3218b08bb57644d78717c"
server_key_version: "encryptionkey@0"
I20260620 07:04:21.489627 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "4202d538077f4483ad230607c23c9d31"
format_stamp: "Formatted at 2026-06-20 07:04:21 on dist-test-slave-wvz3"
server_key: "d625a53f04339cb92f2b6df42e80b4b9"
server_key_iv: "82cd4aca18d3218b08bb57644d78717c"
server_key_version: "encryptionkey@0"
I20260620 07:04:21.494882 20184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:04:21.498548 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.500069 20184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:21.500219 20184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "4202d538077f4483ad230607c23c9d31"
format_stamp: "Formatted at 2026-06-20 07:04:21 on dist-test-slave-wvz3"
server_key: "d625a53f04339cb92f2b6df42e80b4b9"
server_key_iv: "82cd4aca18d3218b08bb57644d78717c"
server_key_version: "encryptionkey@0"
I20260620 07:04:21.500447 20184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:21.522212 20184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:21.526563 20184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:21.526824 20184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:21.527822 20184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:21.529119 20184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:21.529199 20184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.529287 20184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:21.529328 20184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:21.558220 20184 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:40525
I20260620 07:04:21.558265 20311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:40525 every 8 connection(s)
I20260620 07:04:21.559515 20184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:21.561338 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20184
I20260620 07:04:21.561518 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:21.561975 11591 external_mini_cluster.cc:1428] Setting key fc0f8f152e19b693050147de04aa9e93
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939061, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:21.580319 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.562130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:58337 (local address 127.11.81.254:43279)
0620 07:04:21.562336 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:21.562342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:21.563611 (+ 1269us) server_negotiation.cc:407] Connection header received
0620 07:04:21.565452 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:21.565456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:21.565523 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:21.565600 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:21.567557 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.568282 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.570591 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.570984 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.571078 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.574794 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:21.574798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:21.574802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:21.574835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:21.577550 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.578340 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.578343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.578346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.578484 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.578816 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.578818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.578820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.579068 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:21.579177 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:21.579904 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 07:04:21.580068 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:04:21.581259 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.562608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:58337)
0620 07:04:21.563455 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:21.563502 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:21.565205 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:21.565822 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:21.565834 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:21.566556 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:21.567393 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:21.567420 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.568448 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.568453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:21.570451 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:21.570466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.571208 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.571216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:21.571568 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.572470 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:21.572500 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:21.574588 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:21.577696 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:21.577723 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:21.577747 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:21.578236 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:21.578561 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:21.578564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:21.578566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:21.578739 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:21.579158 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:21.579164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:21.579576 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 07:04:21.579804 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 07:04:21.580200 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260620 07:04:21.582566 20312 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43279
I20260620 07:04:21.582846 20312 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:21.583644 20312 heartbeater.cc:511] Master 127.11.81.254:43279 requested a full tablet report, sending...
I20260620 07:04:21.585004 19856 ts_manager.cc:194] Registered new tserver with Master: 4202d538077f4483ad230607c23c9d31 (127.11.81.195:40525)
I20260620 07:04:21.585762 19856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:58337
I20260620 07:04:21.589413 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:04:21.600243 20045 heartbeater.cc:503] Master 127.11.81.254:43279 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
Entry for principal oryx/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab.
WARNING: no policy specified for oryx/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
Entry for principal oryx/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab.
WARNING: no policy specified for oryx/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
Entry for principal oryx/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab.
I20260620 07:04:21.714576 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19917
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:04:21.730172 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.715781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.11.81.193:36241)
0620 07:04:21.716414 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:21.716421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:21.716447 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:04:21.716515 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:21.716522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:21.716744 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:21.716875 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:21.718100 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.719147 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.720353 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.720926 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.721022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.724860 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:21.724873 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:21.724888 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:21.724924 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:21.727733 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.728406 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.728414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.728416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.728557 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.728845 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.728848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.728850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.729188 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:21.729410 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:21.729617 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:04:21.729812 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
I20260620 07:04:21.731421 20040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:21.814123 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19917
I20260620 07:04:21.827595 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20051
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:04:21.843348 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.828457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.11.81.194:42623)
0620 07:04:21.828955 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:21.828963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:21.828990 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:21.829052 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:21.829058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:21.829271 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:21.829402 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:21.830521 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.831559 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.832897 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.833525 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.833622 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.837971 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:21.837985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:21.838002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:21.838040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:21.840783 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.841467 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.841474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.841476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.841588 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.841859 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.841862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.841864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.842217 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:21.842437 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:21.842751 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:04:21.842964 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260620 07:04:21.844578 20174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:21.925779 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20051
I20260620 07:04:21.938481 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20184
Jun 20 07:04:21 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:04:21.954268 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:21.939580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.11.81.195:40525)
0620 07:04:21.940224 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:21.940232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:21.940259 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:21.940317 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:21.940323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:21.940553 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:21.940684 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:21.941943 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.942971 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.944259 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:21.944827 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:21.944920 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:21.949081 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:21.949093 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:21.949108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:21.949147 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:21.951642 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.952330 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.952337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.952339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.952476 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:21.952873 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:21.952876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:21.952878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:21.953255 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:21.953508 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:21.953777 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:04:21.953990 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":327,"threads_started":1}
I20260620 07:04:21.955560 20307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:22.037124 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20184
I20260620 07:04:22.049713 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 19824
I20260620 07:04:22.050927 19886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:22.152386 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 19824
I20260620 07:04:22.169376 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43279
--webserver_interface=127.11.81.254
--webserver_port=34883
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:22.496471 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:22.496794 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:22.496928 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:22.504331 20329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:22.504452 20329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:22.504504 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:22.504562 20329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:22.504604 20329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:22.514623 20329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=34883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:22.516017 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:22.517313 20329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:22.529016 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:22.529137 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:22.529094 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:22.529991 20329 server_base.cc:1034] running on GCE node
I20260620 07:04:22.530761 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:22.532271 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:22.533481 20329 hybrid_clock.cc:648] HybridClock initialized: now 1781939062533430 us; error 49 us; skew 500 ppm
I20260620 07:04:22.533838 20329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:22 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939062, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:22.536968 20329 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:04:22.538583 20329 webserver.cc:460] Webserver started at http://127.11.81.254:34883/ using document root <none> and password file <none>
I20260620 07:04:22.539346 20329 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:22.539491 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:22.545110 20329 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 07:04:22.548120 20344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:22.549542 20329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:22.549695 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "475b732a52494932a2af6cdc40113e64"
format_stamp: "Formatted at 2026-06-20 07:04:19 on dist-test-slave-wvz3"
server_key: "3a54a5c3e94517163eb26eac87fd96d0"
server_key_iv: "3d00365fca3c3b650eddb9ece77b894a"
server_key_version: "encryptionkey@0"
I20260620 07:04:22.550674 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:22.565892 20329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:22.570053 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:22.570312 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:22.587687 20329 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43279
I20260620 07:04:22.587694 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43279 every 8 connection(s)
I20260620 07:04:22.589208 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:22.590430 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20329
I20260620 07:04:22.591413 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:36241
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:43279
--webserver_port=35363
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:04:22.598721 20396 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:04:22.602437 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Bootstrap starting.
I20260620 07:04:22.621788 20396 log.cc:824] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:22.630591 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:04:22.631101 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Bootstrap complete.
I20260620 07:04:22.640323 20396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:22.641194 20396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475b732a52494932a2af6cdc40113e64, State: Initialized, Role: FOLLOWER
I20260620 07:04:22.641825 20396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:22.642043 20396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:22.642154 20396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:22.642309 20396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:04:22.646277 20396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:22.646788 20396 leader_election.cc:302] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475b732a52494932a2af6cdc40113e64; no voters:
I20260620 07:04:22.647401 20396 leader_election.cc:288] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:04:22.647552 20400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:04:22.648547 20400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [term 2 LEADER]: Becoming Leader. State: Replica: 475b732a52494932a2af6cdc40113e64, State: Running, Role: LEADER
I20260620 07:04:22.649202 20400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } }
I20260620 07:04:22.649627 20396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:22.651741 20401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "475b732a52494932a2af6cdc40113e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } } }
I20260620 07:04:22.652130 20401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:22.654377 20402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475b732a52494932a2af6cdc40113e64. Latest consensus state: current_term: 2 leader_uuid: "475b732a52494932a2af6cdc40113e64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475b732a52494932a2af6cdc40113e64" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43279 } } }
I20260620 07:04:22.654726 20402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:22.656381 20406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:22.662778 20406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:22.664451 20406 catalog_manager.cc:1259] Loaded cluster ID: 57f9398a57c94c0c95968b8704042eff
I20260620 07:04:22.664649 20406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:22.669323 20406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:22.671032 20406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 475b732a52494932a2af6cdc40113e64: Loaded TSK: 0
I20260620 07:04:22.672888 20406 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:04:22.928231 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:22.928530 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:22.928651 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:22.936223 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:22.936342 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:22.936486 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:22.946655 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.193.keytab
--principal=oryx/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:36241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=35363
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:22.948094 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:22.949355 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:22.964359 20423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:22.964326 20422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:22.965092 20425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:22.966035 20398 server_base.cc:1034] running on GCE node
I20260620 07:04:22.966674 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:22.967767 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:22.968997 20398 hybrid_clock.cc:648] HybridClock initialized: now 1781939062968937 us; error 48 us; skew 500 ppm
I20260620 07:04:22.969347 20398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:22 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939062, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:22.972570 20398 init.cc:377] Logged in from keytab as oryx/127.11.81.193@KRBTEST.COM (short username oryx)
I20260620 07:04:22.974115 20398 webserver.cc:460] Webserver started at http://127.11.81.193:35363/ using document root <none> and password file <none>
I20260620 07:04:22.974810 20398 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:22.974928 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:22.980353 20398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:04:22.983062 20432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:22.984548 20398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:04:22.984695 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "7443562fa6cf4a7c9f27d95c9e23bb48"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "5a4eba552ebc071da612ccccb7428ecc"
server_key_iv: "cd8d66895e63dda216b2772f5c889c39"
server_key_version: "encryptionkey@0"
I20260620 07:04:22.985812 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:23.010114 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:23.014482 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:23.014763 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:23.015872 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:23.017270 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:23.017371 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.017484 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:23.017563 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.047825 20398 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:36241
I20260620 07:04:23.047931 20544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:36241 every 8 connection(s)
I20260620 07:04:23.049281 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:23.056171 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20398
I20260620 07:04:23.056815 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:42623
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:43279
--webserver_port=39699
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:04:23.387817 20549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:23.388099 20549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:23.388219 20549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:23.395689 20549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:23.395809 20549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:23.395964 20549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:23.405887 20549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.194.keytab
--principal=oryx/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:42623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=39699
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:23.407222 20549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:23.408510 20549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:23.424279 20556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:23.424377 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:23.424525 20558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:23.425675 20549 server_base.cc:1034] running on GCE node
I20260620 07:04:23.426307 20549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:23.427340 20549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:23.428539 20549 hybrid_clock.cc:648] HybridClock initialized: now 1781939063428462 us; error 77 us; skew 500 ppm
I20260620 07:04:23.428845 20549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939063, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:23.432442 20549 init.cc:377] Logged in from keytab as oryx/127.11.81.194@KRBTEST.COM (short username oryx)
I20260620 07:04:23.434612 20549 webserver.cc:460] Webserver started at http://127.11.81.194:39699/ using document root <none> and password file <none>
I20260620 07:04:23.435776 20549 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:23.435971 20549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:23.445120 20549 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 07:04:23.449591 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.451701 20549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:04:23.451948 20549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "71bc01596add4e62a32ad617eef2ed19"
format_stamp: "Formatted at 2026-06-20 07:04:20 on dist-test-slave-wvz3"
server_key: "b285016e5ae5aa73a2fa9e69d0f92c10"
server_key_iv: "0ea135a57f85df2a03d4aab419d7eb6a"
server_key_version: "encryptionkey@0"
I20260620 07:04:23.453652 20549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:23.492959 20549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:23.500820 20549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:23.501230 20549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:23.502697 20549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:23.504967 20549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:23.505131 20549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.505286 20549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:23.505385 20549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.544700 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:42623 every 8 connection(s)
I20260620 07:04:23.544692 20549 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:42623
I20260620 07:04:23.547122 20549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:23.553555 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20549
I20260620 07:04:23.554342 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:40525
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:43279
--webserver_port=43609
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:04:23.895340 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:23.895655 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:23.895808 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:23.903030 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:23.903160 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:23.903357 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:23.913357 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/krb5kdc/oryx_127.11.81.195.keytab
--principal=oryx/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:40525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=43609
--tserver_master_addrs=127.11.81.254:43279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:23.914728 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:23.916082 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:23.932302 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:23.932287 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:23.939267 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:23.939900 20682 server_base.cc:1034] running on GCE node
I20260620 07:04:23.940579 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:23.941622 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:23.942848 20682 hybrid_clock.cc:648] HybridClock initialized: now 1781939063942788 us; error 53 us; skew 500 ppm
I20260620 07:04:23.943166 20682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:23 dist-test-slave-wvz3 krb5kdc[19811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939063, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:23.946429 20682 init.cc:377] Logged in from keytab as oryx/127.11.81.195@KRBTEST.COM (short username oryx)
I20260620 07:04:23.948300 20682 webserver.cc:460] Webserver started at http://127.11.81.195:43609/ using document root <none> and password file <none>
I20260620 07:04:23.949322 20682 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:23.949481 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:23.955459 20682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:04:23.958513 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.960062 20682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:23.960227 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "4202d538077f4483ad230607c23c9d31"
format_stamp: "Formatted at 2026-06-20 07:04:21 on dist-test-slave-wvz3"
server_key: "d625a53f04339cb92f2b6df42e80b4b9"
server_key_iv: "82cd4aca18d3218b08bb57644d78717c"
server_key_version: "encryptionkey@0"
I20260620 07:04:23.961297 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:23.992063 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:23.996394 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:23.996656 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:23.997709 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:23.999132 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:23.999229 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:23.999364 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:23.999434 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:24.041983 20682 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:40525
I20260620 07:04:24.042021 20810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:40525 every 8 connection(s)
I20260620 07:04:24.043550 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:24.049211 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20682
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:24.088724 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.050466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.11.81.254:43279)
0620 07:04:24.061740 (+ 11274us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.061746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.061768 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:04:24.061822 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.061826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.061894 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.062019 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.064487 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.065533 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.070479 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.071509 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.071664 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.075962 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:24.075976 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:24.076006 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:24.076067 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:24.082410 (+ 6343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:24.083596 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:24.083607 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:24.083617 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:24.083843 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:24.084456 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:24.084462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:24.084466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:24.085223 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:24.085565 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:24.087020 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0620 07:04:24.087601 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":11135,"thread_start_us":4968,"threads_started":1}
W20260620 07:04:24.286181 20548 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.223s user 0.007s sys 0.005s
W20260620 07:04:24.288086 20547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:23.052290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:36743 (local address 127.11.81.254:43279)
0620 07:04:23.052940 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:23.052951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:23.053346 (+ 395us) server_negotiation.cc:407] Connection header received
0620 07:04:23.055722 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:23.055735 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:23.056094 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:23.056478 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:23.058163 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.059526 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.061137 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.061815 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.062003 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.287699 (+1225696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:36743: BlockingRecv error: recv got EOF from 127.11.81.193:36743 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":240,"threads_started":1}
I20260620 07:04:24.289280 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:23.052338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:36743)
0620 07:04:23.053183 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:23.053229 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:23.055150 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:23.056668 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:23.056680 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:23.057353 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:23.057966 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:23.057985 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.059718 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.059722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:23.060988 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:23.061002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.062055 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.062063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:23.062324 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:23.062970 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:23.062993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.287314 (+1224321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":400,"threads_started":1}
W20260620 07:04:24.290560 20545 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43279 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.81.254:43279: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:24.304895 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.291534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:33397)
0620 07:04:24.291781 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:24.291806 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:24.291961 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:24.292462 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:24.292467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:24.293013 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:24.293280 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.293294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.294987 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.294994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.296807 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.296831 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.298294 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.298303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.298510 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.299398 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:24.299427 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.304529 (+ 5102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 07:04:24.305361 20547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:24.291867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:33397 (local address 127.11.81.254:43279)
0620 07:04:24.292066 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.292073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.292098 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:04:24.292175 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.292181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.292287 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.292422 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.293479 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.294782 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.297038 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.297790 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.297903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.305044 (+ 7141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:33397: BlockingRecv error: recv got EOF from 127.11.81.193:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:24.319762 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.306735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:33621)
0620 07:04:24.306939 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:24.306964 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:24.307108 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:24.307645 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:24.307650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:24.308130 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:24.308396 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.308410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.310159 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.310166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.311978 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.312001 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.313292 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.313300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.313527 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.314346 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:24.314371 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.319421 (+ 5050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 07:04:24.320444 20545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:04:24.320860 20547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:24.307052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:33621 (local address 127.11.81.254:43279)
0620 07:04:24.307270 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.307276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.307319 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:04:24.307393 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.307399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.307486 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.307603 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.308608 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.309912 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.312246 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.313083 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.313258 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.319735 (+ 6477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:33621: BlockingRecv error: recv got EOF from 127.11.81.193:33621 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:04:24.720804 20681 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.156s user 0.000s sys 0.016s
W20260620 07:04:24.722204 20680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:23.550219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:33753 (local address 127.11.81.254:43279)
0620 07:04:23.550761 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:23.550767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:23.551985 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 07:04:23.555476 (+ 3491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:23.555482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:23.555571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:23.555707 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:23.558247 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.559304 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.562240 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.562847 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.562977 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.721867 (+1158890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:33753: BlockingRecv error: recv got EOF from 127.11.81.194:33753 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":268,"threads_started":1}
I20260620 07:04:24.722733 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:23.550689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:33753)
0620 07:04:23.551754 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:23.551828 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:23.555011 (+ 3183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:23.556063 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:23.556079 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:23.556979 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:23.558004 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:23.558037 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.559503 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.559510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:23.562008 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:23.562031 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:23.563131 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:23.563139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:23.563590 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:23.564478 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:23.564519 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.721658 (+1157139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":376,"threads_started":1}
W20260620 07:04:24.723889 20678 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43279 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.81.254:43279: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:24.736050 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.724620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:55653)
0620 07:04:24.724830 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:24.724855 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:24.724988 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:24.725290 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:24.725295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:24.725815 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:24.726054 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.726067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.727103 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.727109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.728911 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.728930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.729675 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.729682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.729861 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.730603 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:24.730638 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.735645 (+ 5007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:04:24.736104 20680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:24.724734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:55653 (local address 127.11.81.254:43279)
0620 07:04:24.724884 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.724888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.724905 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:04:24.725110 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.725114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.725173 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.725246 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.726211 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.726978 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.729090 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.729504 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.729607 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.735841 (+ 6234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:55653: BlockingRecv error: recv got EOF from 127.11.81.194:55653 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:24 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:24.745695 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.736920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:40737)
0620 07:04:24.737482 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:24.737500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:24.737610 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:24.737924 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:24.737928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:24.738254 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:24.738409 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.738417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.739450 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.739455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.740421 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.740433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.741128 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.741133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.741340 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.742026 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:24.742044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:24.745392 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
W20260620 07:04:24.745874 20680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:24.737101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:40737 (local address 127.11.81.254:43279)
0620 07:04:24.737284 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.737290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.737554 (+ 264us) server_negotiation.cc:407] Connection header received
0620 07:04:24.737720 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.737726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.737802 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.737880 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.738540 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.739312 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.740550 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.740979 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.741083 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.745630 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:40737: BlockingRecv error: recv got EOF from 127.11.81.194:40737 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:04:24.746378 20678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:04:25.109472 20814 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.036s user 0.002s sys 0.013s
W20260620 07:04:25.110707 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:24.046440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:45201 (local address 127.11.81.254:43279)
0620 07:04:24.055799 (+ 9359us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:24.055808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:24.060020 (+ 4212us) server_negotiation.cc:407] Connection header received
0620 07:04:24.060111 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:24.060117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:24.060213 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:24.060324 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:24.066036 (+ 5712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.067468 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.070406 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.071364 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.071529 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.110391 (+1038862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:45201: BlockingRecv error: recv got EOF from 127.11.81.195:45201 (error 108)
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":9174,"spinlock_wait_cycles":27008,"thread_start_us":17251,"threads_started":1}
I20260620 07:04:25.111378 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:24.047156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:45201)
0620 07:04:24.056550 (+ 9394us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:24.056632 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:24.059596 (+ 2964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:24.063693 (+ 4097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:24.063706 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:24.064620 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:24.065634 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.065665 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.067733 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.067739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.070177 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:24.070199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:24.072095 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:24.072103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:24.072488 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:24.073226 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:24.073259 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:25.110215 (+1036956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8976,"thread_start_us":8742,"threads_started":1}
W20260620 07:04:25.112280 20811 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43279 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.81.254:43279: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:25.121152 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:25.112959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:37913)
0620 07:04:25.113156 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:25.113176 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:25.113305 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:25.113602 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:25.113607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:25.113970 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:25.114128 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.114136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.115122 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.115128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.116140 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.116153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.116834 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.116840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.116995 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.117530 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:25.117546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:25.120854 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 07:04:25.121320 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:25.113084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:37913 (local address 127.11.81.254:43279)
0620 07:04:25.113251 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:25.113256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:25.113274 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:04:25.113418 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:25.113422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:25.113481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:25.113559 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:25.114267 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.114989 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.116273 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.116670 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.116786 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.121097 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:37913: BlockingRecv error: recv got EOF from 127.11.81.195:37913 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:25.130079 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:25.121858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:33721)
0620 07:04:25.122012 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:25.122031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:25.122137 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:25.122508 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:25.122513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:25.122904 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:25.123085 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.123094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.124087 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.124092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.125022 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.125034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.125747 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.125760 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.125933 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.126440 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:25.126458 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:25.129802 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:04:25.130312 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:25.121963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33721 (local address 127.11.81.254:43279)
0620 07:04:25.122250 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:25.122267 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:25.122280 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:04:25.122320 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:25.122323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:25.122377 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:25.122453 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:25.123216 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.123952 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.125160 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.125609 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.125712 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.130043 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:33721: BlockingRecv error: recv got EOF from 127.11.81.195:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260620 07:04:25.130442 20811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:25.329638 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:25.321009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:54123)
0620 07:04:25.321568 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:25.321591 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:25.321726 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:25.322067 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:25.322071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:25.322492 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:25.322634 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.322643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.323661 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.323665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.324619 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.324631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.325340 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.325347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.325486 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.326057 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:25.326074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:25.329401 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":302,"threads_started":1}
W20260620 07:04:25.329766 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:25.321109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:54123 (local address 127.11.81.254:43279)
0620 07:04:25.321282 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:25.321288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:25.321670 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:04:25.321843 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:25.321849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:25.321932 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:25.322014 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:25.322767 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.323524 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.324749 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.325157 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.325263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.329584 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:54123: BlockingRecv error: recv got EOF from 127.11.81.193:54123 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:25 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:25.756170 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:25.747016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:39903)
0620 07:04:25.747617 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:25.747633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:25.747749 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:25.748089 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:25.748094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:25.748477 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:25.748624 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.748633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.749902 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.749906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.751054 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:25.751068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.751814 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.751820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:25.751972 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.752519 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:25.752538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:25.755856 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":384,"threads_started":1}
W20260620 07:04:25.756309 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:25.747119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:39903 (local address 127.11.81.254:43279)
0620 07:04:25.747406 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:25.747410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:25.747705 (+ 295us) server_negotiation.cc:407] Connection header received
0620 07:04:25.747862 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:25.747866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:25.747932 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:25.748020 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:25.748733 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.749723 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.751177 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:25.751620 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:25.751725 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:25.756084 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:39903: BlockingRecv error: recv got EOF from 127.11.81.194:39903 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:26.144085 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:26.131004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:34677)
0620 07:04:26.131639 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:26.131661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:26.131814 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:26.132192 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:26.132196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:26.132768 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:26.132916 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.132926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.133994 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.133998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.134933 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.134946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.135866 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.135871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.135993 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.140143 (+ 4150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:26.140162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:26.143760 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":364,"threads_started":1}
W20260620 07:04:26.144176 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:26.131116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:34677 (local address 127.11.81.254:43279)
0620 07:04:26.131366 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:26.131375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:26.131756 (+ 381us) server_negotiation.cc:407] Connection header received
0620 07:04:26.131952 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:26.131958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:26.132023 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:26.132130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:26.133073 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.133853 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.135112 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.135655 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.135779 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.143958 (+ 8179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:34677: BlockingRecv error: recv got EOF from 127.11.81.195:34677 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:26.340340 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:26.330681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:33589)
0620 07:04:26.331672 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:26.331689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:26.331818 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:26.332134 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:26.332140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:26.332568 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:26.332740 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.332750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.333824 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.333830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.334742 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.334754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.335456 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.335462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.335628 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.336208 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:26.336230 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:26.339971 (+ 3741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":769,"threads_started":1}
W20260620 07:04:26.340706 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:26.330765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:33589 (local address 127.11.81.254:43279)
0620 07:04:26.330967 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:26.330973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:26.331777 (+ 804us) server_negotiation.cc:407] Connection header received
0620 07:04:26.331930 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:26.331934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:26.331998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:26.332078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:26.332890 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.333689 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.334875 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.335274 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.335413 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.340254 (+ 4841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:33589: BlockingRecv error: recv got EOF from 127.11.81.193:33589 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:26 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:26.766151 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:26.757183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:36089)
0620 07:04:26.757628 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:26.757644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:26.757779 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:26.758062 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:26.758065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:26.758488 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:26.758664 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.758672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.759730 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.759734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.760858 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:26.760873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.761719 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.761725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:26.761905 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.762484 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:26.762502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:26.765892 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
W20260620 07:04:26.766285 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:26.757271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:36089 (local address 127.11.81.254:43279)
0620 07:04:26.757512 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:26.757518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:26.757725 (+ 207us) server_negotiation.cc:407] Connection header received
0620 07:04:26.757883 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:26.757887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:26.757950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:26.758032 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:26.758803 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.759610 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.760996 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:26.761530 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:26.761661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:26.766079 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:36089: BlockingRecv error: recv got EOF from 127.11.81.194:36089 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:27.154407 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:27.144983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:46047)
0620 07:04:27.145630 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:27.145648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:27.145786 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:27.146152 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:27.146156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:27.146655 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:27.146857 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.146876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.148055 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.148059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.149009 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.149023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.149814 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.149818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.149946 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.150549 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:27.150568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:27.154080 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":402,"threads_started":1}
W20260620 07:04:27.154803 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:27.145184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:46047 (local address 127.11.81.254:43279)
0620 07:04:27.145413 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:27.145418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:27.145758 (+ 340us) server_negotiation.cc:407] Connection header received
0620 07:04:27.145920 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:27.145924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:27.146008 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:27.146102 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:27.147038 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.147912 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.149160 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.149641 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.149746 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.154294 (+ 4548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:46047: BlockingRecv error: recv got EOF from 127.11.81.195:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:27.351351 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:27.341277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:40749)
0620 07:04:27.341864 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:27.341883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:27.342034 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:27.342326 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:27.342328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:27.342720 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:27.342881 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.342890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.343920 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.343925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.344932 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.344946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.345664 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.345669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.345828 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.346396 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:27.346414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:27.350983 (+ 4569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":338,"threads_started":1}
W20260620 07:04:27.351537 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:27.341352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:40749 (local address 127.11.81.254:43279)
0620 07:04:27.341562 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:27.341568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:27.341963 (+ 395us) server_negotiation.cc:407] Connection header received
0620 07:04:27.342135 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:27.342138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:27.342199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:27.342298 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:27.342996 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.343768 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.345040 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.345460 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.345551 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.351318 (+ 5767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:40749: BlockingRecv error: recv got EOF from 127.11.81.193:40749 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:27 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:27.775868 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:27.767237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:39081)
0620 07:04:27.767794 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:27.767811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:27.767925 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:27.768268 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:27.768272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:27.768676 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:27.768842 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.768851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.769840 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.769843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.770761 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:27.770772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.771411 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.771414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:27.771585 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.772113 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:27.772129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:27.775632 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":270,"threads_started":1}
W20260620 07:04:27.776042 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:27.767352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:39081 (local address 127.11.81.254:43279)
0620 07:04:27.767549 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:27.767555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:27.767891 (+ 336us) server_negotiation.cc:407] Connection header received
0620 07:04:27.768021 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:27.768025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:27.768089 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:27.768183 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:27.768963 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.769676 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.770867 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:27.771296 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:27.771403 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:27.775835 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:39081: BlockingRecv error: recv got EOF from 127.11.81.194:39081 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:28.164304 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:28.155436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:36129)
0620 07:04:28.156084 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:28.156103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:28.156236 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:28.156541 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:28.156546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:28.156900 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:28.157071 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.157080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.158154 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.158159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.159130 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.159142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.159926 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.159930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.160096 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.160777 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:28.160796 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:28.164059 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":345,"threads_started":1}
W20260620 07:04:28.164578 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:28.155510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:36129 (local address 127.11.81.254:43279)
0620 07:04:28.155708 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:28.155716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:28.156169 (+ 453us) server_negotiation.cc:407] Connection header received
0620 07:04:28.156351 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:28.156355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:28.156428 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:28.156508 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:28.157228 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.158014 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.159264 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.159770 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.159876 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.164316 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:36129: BlockingRecv error: recv got EOF from 127.11.81.195:36129 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:28.361245 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:28.352384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:42267)
0620 07:04:28.352994 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:28.353014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:28.353151 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:28.353503 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:28.353507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:28.353920 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:28.354091 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.354100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.355101 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.355106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.356139 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.356151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.356855 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.356860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.357046 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.357586 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:28.357602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:28.360895 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":363,"threads_started":1}
W20260620 07:04:28.361404 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:28.352517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:42267 (local address 127.11.81.254:43279)
0620 07:04:28.352730 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:28.352736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:28.353100 (+ 364us) server_negotiation.cc:407] Connection header received
0620 07:04:28.353283 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:28.353287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:28.353348 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:28.353437 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:28.354245 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.354966 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.356278 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.356722 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.356827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.361161 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:42267: BlockingRecv error: recv got EOF from 127.11.81.193:42267 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:28 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:28.785800 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:28.776859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:45599)
0620 07:04:28.777387 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:28.777404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:28.777518 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:28.777827 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:28.777831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:28.778236 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:28.778405 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.778414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.779497 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.779501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.780527 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:28.780540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.781250 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.781254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:28.781368 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.781909 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:28.781926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:28.785412 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":294,"threads_started":1}
W20260620 07:04:28.785859 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:28.776989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:45599 (local address 127.11.81.254:43279)
0620 07:04:28.777211 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:28.777217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:28.777487 (+ 270us) server_negotiation.cc:407] Connection header received
0620 07:04:28.777608 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:28.777611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:28.777672 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:28.777753 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:28.778565 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.779351 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.780665 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:28.781132 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:28.781228 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:28.785642 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:45599: BlockingRecv error: recv got EOF from 127.11.81.194:45599 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:29.174558 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:29.165369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:40429)
0620 07:04:29.165869 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:29.165885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:29.166022 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:29.166350 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:29.166353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:29.166742 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:29.166921 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.166929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.168044 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.168049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.169361 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.169374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.170052 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.170057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.170200 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.170865 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:29.170882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:29.174265 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
W20260620 07:04:29.174700 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:29.165404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:40429 (local address 127.11.81.254:43279)
0620 07:04:29.165598 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:29.165606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:29.165965 (+ 359us) server_negotiation.cc:407] Connection header received
0620 07:04:29.166156 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:29.166160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:29.166232 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:29.166309 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:29.167043 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.167906 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.169490 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.169902 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.170014 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.174478 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:40429: BlockingRecv error: recv got EOF from 127.11.81.195:40429 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:29.371429 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:29.362390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:34783)
0620 07:04:29.362943 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:29.362959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:29.363127 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:29.363555 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:29.363560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:29.363982 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:29.364130 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.364138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.365156 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.365161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.366264 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.366279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.367063 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.367068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.367225 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.367843 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:29.367861 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:29.371180 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
W20260620 07:04:29.371690 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:29.362540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:34783 (local address 127.11.81.254:43279)
0620 07:04:29.362779 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:29.362786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:29.363042 (+ 256us) server_negotiation.cc:407] Connection header received
0620 07:04:29.363258 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:29.363263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:29.363365 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:29.363452 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:29.364281 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.365027 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.366421 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.366905 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.367014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.371406 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:34783: BlockingRecv error: recv got EOF from 127.11.81.193:34783 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:29 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:29.795320 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:29.786782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:43537)
0620 07:04:29.787326 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:29.787345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:29.787472 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:29.787847 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:29.787852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:29.788291 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:29.788435 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.788445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.789484 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.789490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.790414 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:29.790426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.791123 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.791129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:29.791352 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.791897 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:29.791914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:29.794959 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":253,"threads_started":1}
W20260620 07:04:29.795434 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:29.786968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:43537 (local address 127.11.81.254:43279)
0620 07:04:29.787153 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:29.787158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:29.787419 (+ 261us) server_negotiation.cc:407] Connection header received
0620 07:04:29.787580 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:29.787585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:29.787705 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:29.787799 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:29.788578 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.789360 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.790542 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:29.790983 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:29.791089 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:29.795173 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:43537: BlockingRecv error: recv got EOF from 127.11.81.194:43537 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:30.183900 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:30.175486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:36073)
0620 07:04:30.175987 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:30.176004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:30.176118 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:30.176414 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:30.176417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:30.176837 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:30.177001 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.177010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.177951 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.177955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.178857 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.178867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.179577 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.179581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.179711 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.180243 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:30.180259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:30.183583 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":267,"threads_started":1}
W20260620 07:04:30.184087 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:30.175594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:36073 (local address 127.11.81.254:43279)
0620 07:04:30.175806 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:30.175812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:30.176072 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:04:30.176215 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:30.176219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:30.176291 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:30.176371 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:30.177123 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.177825 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.178989 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.179452 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.179544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.183821 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:36073: BlockingRecv error: recv got EOF from 127.11.81.195:36073 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:30.381300 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:30.372315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:36783)
0620 07:04:30.372813 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:30.372830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:30.372980 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:30.373323 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:30.373328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:30.373733 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:30.373900 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.373909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.374906 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.374911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.375913 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.375927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.376617 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.376623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.376782 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.377322 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:30.377338 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:30.380993 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
W20260620 07:04:30.381479 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:30.372399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:36783 (local address 127.11.81.254:43279)
0620 07:04:30.372618 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:30.372623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:30.372956 (+ 333us) server_negotiation.cc:407] Connection header received
0620 07:04:30.373123 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:30.373126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:30.373187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:30.373270 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:30.374041 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.374771 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.376053 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.376468 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.376576 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.381207 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:36783: BlockingRecv error: recv got EOF from 127.11.81.193:36783 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:30 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:30.805955 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:30.796417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:43871)
0620 07:04:30.796903 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:30.796918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:30.797168 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:30.797594 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:30.797599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:30.798029 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:30.798212 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.798223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.799236 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.799241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.800213 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:30.800226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.800942 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.800948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:30.801107 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.801677 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:30.801694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:30.805597 (+ 3903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
W20260620 07:04:30.806039 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:30.796512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:43871 (local address 127.11.81.254:43279)
0620 07:04:30.796700 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:30.796706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:30.797095 (+ 389us) server_negotiation.cc:407] Connection header received
0620 07:04:30.797324 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:30.797330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:30.797440 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:30.797537 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:30.798360 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.799086 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.800351 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:30.800784 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:30.800899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:30.805838 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:43871: BlockingRecv error: recv got EOF from 127.11.81.194:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:31.198311 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:31.185143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:44171)
0620 07:04:31.185601 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:31.185621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:31.185748 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:31.186075 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:31.186078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:31.186454 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:31.186615 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.186622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.187800 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.187804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.188723 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.188735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.189459 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.189464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.189643 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.194508 (+ 4865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:31.194529 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:31.198065 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":243,"threads_started":1}
W20260620 07:04:31.198673 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:31.185286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:44171 (local address 127.11.81.254:43279)
0620 07:04:31.185508 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:31.185513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:31.185763 (+ 250us) server_negotiation.cc:407] Connection header received
0620 07:04:31.185831 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:31.185835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:31.185933 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:31.186022 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:31.186785 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.187671 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.188892 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.189297 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.189419 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.198342 (+ 8923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:44171: BlockingRecv error: recv got EOF from 127.11.81.195:44171 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:31.391753 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:31.382563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:40855)
0620 07:04:31.383202 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:31.383220 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:31.383422 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:31.383713 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:31.383716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:31.384150 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:31.384343 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.384354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.385306 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.385310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.386307 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.386320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.387010 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.387015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.387137 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.387772 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:31.387790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:31.391505 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":298,"threads_started":1}
W20260620 07:04:31.391839 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:31.382671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:40855 (local address 127.11.81.254:43279)
0620 07:04:31.382829 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:31.382834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:31.383310 (+ 476us) server_negotiation.cc:407] Connection header received
0620 07:04:31.383526 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:31.383529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:31.383597 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:31.383677 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:31.384477 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.385189 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.386415 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.386827 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.386914 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.391676 (+ 4762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:40855: BlockingRecv error: recv got EOF from 127.11.81.193:40855 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:31 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:31.815790 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:31.806915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:53099)
0620 07:04:31.807738 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:31.807756 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:31.807898 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:31.808264 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:31.808269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:31.808809 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:31.809029 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.809040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.810056 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.810060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.810977 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:31.810988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.811694 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.811698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:31.811816 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.812307 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:31.812323 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:31.815538 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":502,"threads_started":1}
W20260620 07:04:31.815979 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:31.806966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:53099 (local address 127.11.81.254:43279)
0620 07:04:31.807258 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:31.807265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:31.807837 (+ 572us) server_negotiation.cc:407] Connection header received
0620 07:04:31.808024 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:31.808030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:31.808111 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:31.808218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:31.809174 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.809941 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.811110 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:31.811572 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:31.811666 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:31.815745 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:53099: BlockingRecv error: recv got EOF from 127.11.81.194:53099 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:32.208580 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:32.199440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:33027)
0620 07:04:32.200041 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:32.200061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:32.200202 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:32.200560 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:32.200563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:32.201057 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:32.201271 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.201281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.202226 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.202229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.203180 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.203192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.203888 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.203892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.204023 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.204664 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:32.204683 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:32.208258 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":260,"threads_started":1}
W20260620 07:04:32.208678 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:32.199580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33027 (local address 127.11.81.254:43279)
0620 07:04:32.199764 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:32.199770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:32.200152 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:04:32.200315 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:32.200319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:32.200395 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:32.200496 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:32.201404 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.202122 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.203349 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.203759 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.203856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.208479 (+ 4623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:33027: BlockingRecv error: recv got EOF from 127.11.81.195:33027 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:32.402107 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:32.392688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:42529)
0620 07:04:32.393302 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:32.393319 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:32.393447 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:32.393804 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:32.393808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:32.394207 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:32.394370 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.394379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.395492 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.395497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.396478 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.396491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.397284 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.397289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.397500 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.398142 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:32.398160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:32.401847 (+ 3687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":371,"threads_started":1}
W20260620 07:04:32.402287 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:32.392932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:42529 (local address 127.11.81.254:43279)
0620 07:04:32.393164 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:32.393170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:32.393427 (+ 257us) server_negotiation.cc:407] Connection header received
0620 07:04:32.393609 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:32.393614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:32.393679 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:32.393768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:32.394534 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.395320 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.396640 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.397081 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.397218 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.402063 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:42529: BlockingRecv error: recv got EOF from 127.11.81.193:42529 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:32 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:32.826040 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:32.816711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:54253)
0620 07:04:32.817256 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:32.817272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:32.817408 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:32.817867 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:32.817873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:32.818262 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:32.818440 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.818450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.819912 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.819916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.820836 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:32.820848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.821547 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.821553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:32.821696 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.822275 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:32.822294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:32.825769 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
W20260620 07:04:32.826227 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:32.816841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:54253 (local address 127.11.81.254:43279)
0620 07:04:32.817065 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:32.817072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:32.817352 (+ 280us) server_negotiation.cc:407] Connection header received
0620 07:04:32.817530 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:32.817536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:32.817672 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:32.817781 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:32.818579 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.819759 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.820964 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:32.821378 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:32.821482 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:32.826001 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:54253: BlockingRecv error: recv got EOF from 127.11.81.194:54253 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:33.218647 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:33.209513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:59437)
0620 07:04:33.210011 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:33.210028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:33.210149 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:33.210618 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:33.210622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:33.211059 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:33.211234 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.211244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.212369 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.212373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.213343 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.213356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.214176 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.214181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.214300 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.214905 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:33.214922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:33.218396 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
W20260620 07:04:33.218853 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:33.209584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:59437 (local address 127.11.81.254:43279)
0620 07:04:33.209790 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:33.209796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:33.210131 (+ 335us) server_negotiation.cc:407] Connection header received
0620 07:04:33.210353 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:33.210357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:33.210431 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:33.210534 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:33.211443 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.212225 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.213510 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.213995 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.214110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.218609 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:59437: BlockingRecv error: recv got EOF from 127.11.81.195:59437 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:33.412223 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:33.403169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:56523)
0620 07:04:33.403730 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:33.403747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:33.403891 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:33.404262 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:33.404265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:33.404735 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:33.404894 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.404903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.406050 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.406055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.407000 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.407011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.407873 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.407879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.408031 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.408593 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:33.408611 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:33.411990 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":325,"threads_started":1}
W20260620 07:04:33.412487 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:33.403276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:56523 (local address 127.11.81.254:43279)
0620 07:04:33.403518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:33.403524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:33.403835 (+ 311us) server_negotiation.cc:407] Connection header received
0620 07:04:33.404051 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:33.404056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:33.404122 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:33.404215 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:33.405038 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.405911 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.407138 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.407670 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.407780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.412207 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:56523: BlockingRecv error: recv got EOF from 127.11.81.193:56523 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:33 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:33.836440 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:33.827078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:44407)
0620 07:04:33.827629 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:33.827650 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:33.827782 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:33.828116 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:33.828121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:33.828561 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:33.828726 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.828739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.830070 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.830076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.831407 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:33.831423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.832118 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.832124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:33.832270 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.832793 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:33.832810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:33.836124 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
W20260620 07:04:33.836603 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:33.827241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:44407 (local address 127.11.81.254:43279)
0620 07:04:33.827439 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:33.827443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:33.827732 (+ 289us) server_negotiation.cc:407] Connection header received
0620 07:04:33.827887 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:33.827891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:33.827958 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:33.828073 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:33.828929 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.829922 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.831541 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:33.831950 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:33.832049 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:33.836359 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:44407: BlockingRecv error: recv got EOF from 127.11.81.194:44407 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:34.228425 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:34.219757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:42223)
0620 07:04:34.220264 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:34.220280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:34.220401 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:34.220715 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:34.220718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:34.221099 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:34.221247 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.221254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.222205 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.222209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.223182 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.223193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.224111 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.224117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.224275 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.224877 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:34.224901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:34.228190 (+ 3289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
W20260620 07:04:34.228647 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:34.219863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:42223 (local address 127.11.81.254:43279)
0620 07:04:34.220122 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:34.220128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:34.220366 (+ 238us) server_negotiation.cc:407] Connection header received
0620 07:04:34.220498 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:34.220501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:34.220579 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:34.220670 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:34.221361 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.222077 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.223351 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.223946 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.224058 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.228418 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:42223: BlockingRecv error: recv got EOF from 127.11.81.195:42223 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:34.421383 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:34.413074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:59747)
0620 07:04:34.413524 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:34.413540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:34.413654 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:34.413939 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:34.413942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:34.414310 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:34.414462 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.414471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.415456 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.415460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.416466 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.416479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.417138 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.417142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.417249 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.417780 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:34.417795 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:34.421089 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
W20260620 07:04:34.421482 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:34.413162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:59747 (local address 127.11.81.254:43279)
0620 07:04:34.413349 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:34.413355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:34.413611 (+ 256us) server_negotiation.cc:407] Connection header received
0620 07:04:34.413740 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:34.413743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:34.413820 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:34.413900 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:34.414575 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.415319 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.416588 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.417014 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.417112 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.421255 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:59747: BlockingRecv error: recv got EOF from 127.11.81.193:59747 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:34 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:34.846308 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:34.837410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:38481)
0620 07:04:34.837915 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:34.837932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:34.838058 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:34.838377 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:34.838381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:34.838750 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:34.838923 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.838932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.840280 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.840284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.841279 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:34.841291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.841979 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.841984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:34.842112 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.842751 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:34.842772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:34.846072 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":244,"threads_started":1}
W20260620 07:04:34.846576 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:34.837572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:38481 (local address 127.11.81.254:43279)
0620 07:04:34.837859 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:34.837866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:34.838032 (+ 166us) server_negotiation.cc:407] Connection header received
0620 07:04:34.838156 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:34.838160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:34.838225 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:34.838309 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:34.839094 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.840144 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.841423 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:34.841816 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:34.841915 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:34.846294 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:38481: BlockingRecv error: recv got EOF from 127.11.81.194:38481 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:35.238031 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:35.229426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:56751)
0620 07:04:35.229916 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:35.229932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:35.230046 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:35.230381 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:35.230385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:35.230831 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:35.231030 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.231045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.232130 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.232134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.233149 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.233162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.233855 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.233860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.233977 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.234554 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:35.234572 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:35.237801 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
W20260620 07:04:35.238241 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:35.229557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:56751 (local address 127.11.81.254:43279)
0620 07:04:35.229756 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:35.229761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:35.230001 (+ 240us) server_negotiation.cc:407] Connection header received
0620 07:04:35.230134 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:35.230138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:35.230214 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:35.230301 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:35.231196 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.231968 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.233288 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.233711 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.233809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.237999 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:56751: BlockingRecv error: recv got EOF from 127.11.81.195:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:35.431775 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:35.422606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:57519)
0620 07:04:35.423107 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:35.423123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:35.423262 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:35.423673 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:35.423677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:35.424081 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:35.424271 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.424281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.425440 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.425444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.426472 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.426487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.427338 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.427345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.427513 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.428077 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:35.428094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:35.431458 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":274,"threads_started":1}
W20260620 07:04:35.431876 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:35.422735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:57519 (local address 127.11.81.254:43279)
0620 07:04:35.422917 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:35.422922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:35.423201 (+ 279us) server_negotiation.cc:407] Connection header received
0620 07:04:35.423379 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:35.423383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:35.423444 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:35.423530 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:35.424447 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.425254 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.426614 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.427147 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.427266 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.431667 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:57519: BlockingRecv error: recv got EOF from 127.11.81.193:57519 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:35 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:35.855594 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:35.847193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:42493)
0620 07:04:35.847693 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:35.847710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:35.847819 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:35.848072 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:35.848075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:35.848524 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:35.848693 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.848702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.849764 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.849768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.850661 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:35.850695 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.851399 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.851403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:35.851523 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.852094 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:35.852110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:35.855360 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":292,"threads_started":1}
W20260620 07:04:35.855858 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:35.847229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:42493 (local address 127.11.81.254:43279)
0620 07:04:35.847416 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:35.847421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:35.847775 (+ 354us) server_negotiation.cc:407] Connection header received
0620 07:04:35.847903 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:35.847907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:35.847965 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:35.848042 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:35.848849 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.849646 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.850840 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:35.851244 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:35.851396 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:35.855600 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:42493: BlockingRecv error: recv got EOF from 127.11.81.194:42493 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:36.252249 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:36.239062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:35173)
0620 07:04:36.239641 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:36.239662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:36.239787 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:36.240094 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:36.240097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:36.240538 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:36.240730 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.240740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.241826 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.241831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.242979 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.242992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.243838 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.243843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.243977 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.248394 (+ 4417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:36.248415 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:36.252020 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":368,"threads_started":1}
W20260620 07:04:36.252470 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:36.239187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:35173 (local address 127.11.81.254:43279)
0620 07:04:36.239386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:36.239391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:36.239746 (+ 355us) server_negotiation.cc:407] Connection header received
0620 07:04:36.239888 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:36.239892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:36.239959 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:36.240046 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:36.240862 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.241691 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.243102 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.243656 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.243754 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.252261 (+ 8507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:35173: BlockingRecv error: recv got EOF from 127.11.81.195:35173 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:36.441874 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:36.432804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:38093)
0620 07:04:36.433431 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:36.433450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:36.433597 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:36.433968 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:36.433973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:36.434446 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:36.434610 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.434620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.435697 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.435702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.436642 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.436655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.437426 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.437432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.437589 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.438189 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:36.438208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:36.441622 (+ 3414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":344,"threads_started":1}
W20260620 07:04:36.442162 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:36.432986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:38093 (local address 127.11.81.254:43279)
0620 07:04:36.433236 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:36.433242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:36.433562 (+ 320us) server_negotiation.cc:407] Connection header received
0620 07:04:36.433749 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:36.433755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:36.433830 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:36.433914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:36.434759 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.435557 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.436777 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.437254 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.437366 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.441873 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:38093: BlockingRecv error: recv got EOF from 127.11.81.193:38093 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:36 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:36.865272 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:36.856649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:37053)
0620 07:04:36.857161 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:36.857177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:36.857317 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:36.857658 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:36.857662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:36.858170 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:36.858360 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.858370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.859388 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.859391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.860372 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:36.860384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.861097 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.861102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:36.861224 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.861823 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:36.861840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:36.865033 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
W20260620 07:04:36.865504 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:36.856725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:37053 (local address 127.11.81.254:43279)
0620 07:04:36.856905 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:36.856911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:36.857250 (+ 339us) server_negotiation.cc:407] Connection header received
0620 07:04:36.857431 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:36.857435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:36.857516 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:36.857617 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:36.858481 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.859209 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.860521 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:36.860958 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:36.861059 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:36.865243 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:37053: BlockingRecv error: recv got EOF from 127.11.81.194:37053 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:37.262447 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:37.253383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:39269)
0620 07:04:37.253925 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:37.253941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:37.254072 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:37.254374 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:37.254378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:37.254767 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:37.254978 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.254986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.256037 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.256042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.257054 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.257066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.257770 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.257776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.257943 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.258653 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:37.258671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:37.262175 (+ 3504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":286,"threads_started":1}
W20260620 07:04:37.262538 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:37.253453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:39269 (local address 127.11.81.254:43279)
0620 07:04:37.253625 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:37.253632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:37.254026 (+ 394us) server_negotiation.cc:407] Connection header received
0620 07:04:37.254183 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:37.254188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:37.254246 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:37.254325 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:37.255115 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.255907 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.257178 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.257588 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.257719 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.262331 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:39269: BlockingRecv error: recv got EOF from 127.11.81.195:39269 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:37.451594 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:37.442809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:44983)
0620 07:04:37.443262 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:37.443306 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:37.443435 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:37.443715 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:37.443718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:37.444098 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:37.444243 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.444251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.445322 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.445326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.446217 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.446227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.446922 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.446928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.447093 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.447728 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:37.447751 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:37.451358 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
W20260620 07:04:37.451869 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:37.442952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:44983 (local address 127.11.81.254:43279)
0620 07:04:37.443110 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:37.443114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:37.443381 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:04:37.443524 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:37.443528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:37.443588 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:37.443671 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:37.444359 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.445209 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.446322 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.446740 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.446856 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.451609 (+ 4753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:44983: BlockingRecv error: recv got EOF from 127.11.81.193:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:37 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:37.875793 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:37.866209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:49471)
0620 07:04:37.866711 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:37.866730 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:37.866871 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:37.867184 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:37.867189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:37.867725 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:37.867913 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.867925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.869009 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.869015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.870400 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:37.870415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.871165 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.871169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:37.871357 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.871882 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:37.871899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:37.875479 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":255,"threads_started":1}
W20260620 07:04:37.875937 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:37.866429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:49471 (local address 127.11.81.254:43279)
0620 07:04:37.866614 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:37.866621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:37.866799 (+ 178us) server_negotiation.cc:407] Connection header received
0620 07:04:37.866980 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:37.866985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:37.867063 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:37.867161 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:37.868050 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.868863 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.870551 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:37.871036 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:37.871140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:37.875690 (+ 4550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:49471: BlockingRecv error: recv got EOF from 127.11.81.194:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.195@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:38.272805 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:38.263558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.195:39757)
0620 07:04:38.264041 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:38.264056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:38.264189 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:38.264543 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:38.264548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:38.264985 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:38.265134 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.265143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.266189 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.266194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.267146 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.267160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.267855 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.267860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.268021 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.268620 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:38.268639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:38.272515 (+ 3876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
W20260620 07:04:38.272935 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:38.263646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:39757 (local address 127.11.81.254:43279)
0620 07:04:38.263845 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:38.263852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:38.264135 (+ 283us) server_negotiation.cc:407] Connection header received
0620 07:04:38.264330 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:38.264337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:38.264411 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:38.264496 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:38.265310 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.266050 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.267318 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.267712 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.267825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.272739 (+ 4914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.195:39757: BlockingRecv error: recv got EOF from 127.11.81.195:39757 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.193@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:38.461875 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:38.452830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.193:55413)
0620 07:04:38.453329 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:38.453348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:38.453490 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:38.453821 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:38.453825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:38.454246 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:38.454450 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.454459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.455688 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.455692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.456794 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.456806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.457498 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.457504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.457661 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.458239 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:38.458258 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:38.461516 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":217,"threads_started":1}
W20260620 07:04:38.462038 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:38.453000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:55413 (local address 127.11.81.254:43279)
0620 07:04:38.453351 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:38.453356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:38.453427 (+ 71us) server_negotiation.cc:407] Connection header received
0620 07:04:38.453612 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:38.453616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:38.453675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:38.453754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:38.454589 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.455541 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.456955 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.457366 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.457471 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.461744 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.193:55413: BlockingRecv error: recv got EOF from 127.11.81.193:55413 (error 108)
Metrics: {"server-negotiator.queue_time_us":185}
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:04:38 dist-test-slave-wvz3 krb5kdc[19811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.81.194@KRBTEST.COM for oryx/127.11.81.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:04:38.885424 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:38.877105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43279 (local address 127.11.81.194:46219)
0620 07:04:38.877535 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:38.877552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:38.877683 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:38.877943 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:38.877946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:38.878281 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:38.878462 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.878471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.879414 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.879417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.880333 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:38.880344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.880954 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.880957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:38.881077 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.881586 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:38.881602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:38.885082 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.81.254:43279: Server oryx/127.11.81.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
W20260620 07:04:38.885524 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:38.877279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46219 (local address 127.11.81.254:43279)
0620 07:04:38.877489 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:38.877495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:38.877620 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:04:38.877788 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:38.877792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:38.877852 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:38.877929 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:38.878577 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.879270 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.880444 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:38.880849 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:38.880938 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:38.885318 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.81.194:46219: BlockingRecv error: recv got EOF from 127.11.81.194:46219 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:04:39.054010 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20398
I20260620 07:04:39.066107 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:39.055031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.11.81.193:36241)
0620 07:04:39.055741 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:39.055748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:39.055775 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:04:39.055838 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:39.055845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:39.056067 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:39.056277 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:39.057699 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.058759 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.059983 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.060546 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.060685 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:39.062991 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:39.063000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:39.063015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:39.063055 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:39.065590 (+ 2535us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:04:39.065892 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49286: Request ticket server kudu/127.11.81.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":415,"threads_started":1}
W20260620 07:04:39.066202 20861 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49286: Request ticket server kudu/127.11.81.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:04:39.066313 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20398: Not authorized: Client connection negotiation failed: client connection to 127.11.81.193:36241: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:04:39.066552 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20398
I20260620 07:04:39.080603 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20549
I20260620 07:04:39.093374 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:39.081631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.11.81.194:42623)
0620 07:04:39.082258 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:39.082266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:39.082297 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:04:39.082362 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:39.082368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:39.082605 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:39.082772 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:39.084372 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.085478 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.086652 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.087258 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.087394 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:39.090028 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:39.090037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:39.090052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:39.090104 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:39.092819 (+ 2715us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:04:39.093113 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54174: Request ticket server kudu/127.11.81.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":276,"threads_started":1}
W20260620 07:04:39.093477 20862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54174: Request ticket server kudu/127.11.81.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:04:39.093571 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20549: Not authorized: Client connection negotiation failed: client connection to 127.11.81.194:42623: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:04:39.093739 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20549
I20260620 07:04:39.106902 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20682
I20260620 07:04:39.119344 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:39.108051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.11.81.195:40525)
0620 07:04:39.108760 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:39.108769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:39.108801 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:04:39.108876 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:39.108885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:39.109190 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:39.109416 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:39.110673 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.111764 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.113030 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.113578 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.113680 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:39.116286 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:39.116295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:39.116311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:39.116348 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:39.118768 (+ 2420us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:04:39.119060 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42338: Request ticket server kudu/127.11.81.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":346,"threads_started":1}
W20260620 07:04:39.119452 20863 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42338: Request ticket server kudu/127.11.81.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:04:39.119481 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20682: Not authorized: Client connection negotiation failed: client connection to 127.11.81.195:40525: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.81.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:04:39.119674 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20682
I20260620 07:04:39.132571 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20329
I20260620 07:04:39.133989 20391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:39.246559 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20329
2026-06-20T07:04:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20958 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T07:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:39Z Disabled control of system clock
I20260620 07:04:39.316246 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44985
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:44985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:04:39.647540 20868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:39.647869 20868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:39.647954 20868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:39.655400 20868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:39.655540 20868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:39.655593 20868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:39.655656 20868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:39.655706 20868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:39.665148 20868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:44985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:39.666545 20868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:39.668354 20868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:39.680289 20874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:39.680253 20876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:39.680331 20873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:39.680616 20868 server_base.cc:1034] running on GCE node
I20260620 07:04:39.681598 20868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:39.682983 20868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:39.684195 20868 hybrid_clock.cc:648] HybridClock initialized: now 1781939079684141 us; error 41 us; skew 500 ppm
I20260620 07:04:39.684547 20868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:39.686914 20868 webserver.cc:460] Webserver started at http://127.11.81.254:38131/ using document root <none> and password file <none>
I20260620 07:04:39.687682 20868 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:39.687788 20868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:39.688094 20868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:39.691061 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "cd006eaf3082450a908ce775137b3366"
format_stamp: "Formatted at 2026-06-20 07:04:39 on dist-test-slave-wvz3"
server_key: "47055e7a9969044d651666a1a6caa4c4"
server_key_iv: "8070cf3306098aa4390553469be77a01"
server_key_version: "encryptionkey@0"
I20260620 07:04:39.691924 20868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "cd006eaf3082450a908ce775137b3366"
format_stamp: "Formatted at 2026-06-20 07:04:39 on dist-test-slave-wvz3"
server_key: "47055e7a9969044d651666a1a6caa4c4"
server_key_iv: "8070cf3306098aa4390553469be77a01"
server_key_version: "encryptionkey@0"
I20260620 07:04:39.697269 20868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:39.700810 20882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:39.702203 20868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:39.702373 20868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "cd006eaf3082450a908ce775137b3366"
format_stamp: "Formatted at 2026-06-20 07:04:39 on dist-test-slave-wvz3"
server_key: "47055e7a9969044d651666a1a6caa4c4"
server_key_iv: "8070cf3306098aa4390553469be77a01"
server_key_version: "encryptionkey@0"
I20260620 07:04:39.702580 20868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:39.726863 20868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:39.727825 20868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:39.728108 20868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:39.745611 20868 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:44985
I20260620 07:04:39.745630 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:44985 every 8 connection(s)
I20260620 07:04:39.747040 20868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:39.747550 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20868
I20260620 07:04:39.747756 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:39.748525 11591 external_mini_cluster.cc:1428] Setting key 6d2f7450b3432e674f3c4c8b8ce08eee
I20260620 07:04:39.751577 20934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:39.762434 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:39.750675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.11.81.254:44985)
0620 07:04:39.751690 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:39.751705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:39.751753 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:04:39.753345 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:39.753368 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:39.753853 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:39.754230 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:39.754888 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.756251 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.758059 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:39.758732 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:39.758861 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:39.760090 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:39.760097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:39.760115 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:39.760694 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:39.760776 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:39.760800 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:39.760894 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:39.761107 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:04:39.761454 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260620 07:04:39.763816 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366: Bootstrap starting.
I20260620 07:04:39.766947 20934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:39.768004 20934 log.cc:824] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:39.771410 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366: No bootstrap required, opened a new log
I20260620 07:04:39.778815 20934 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } }
I20260620 07:04:39.779194 20934 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:39.779307 20934 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd006eaf3082450a908ce775137b3366, State: Initialized, Role: FOLLOWER
I20260620 07:04:39.779850 20934 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } }
I20260620 07:04:39.780059 20934 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:39.780192 20934 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:39.780341 20934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:39.782437 20934 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } }
I20260620 07:04:39.782917 20934 leader_election.cc:302] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd006eaf3082450a908ce775137b3366; no voters:
I20260620 07:04:39.783520 20934 leader_election.cc:288] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:39.783641 20939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:39.784782 20939 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [term 1 LEADER]: Becoming Leader. State: Replica: cd006eaf3082450a908ce775137b3366, State: Running, Role: LEADER
I20260620 07:04:39.785287 20939 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } }
I20260620 07:04:39.785588 20934 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:39.787846 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd006eaf3082450a908ce775137b3366. Latest consensus state: current_term: 1 leader_uuid: "cd006eaf3082450a908ce775137b3366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } } }
I20260620 07:04:39.788323 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd006eaf3082450a908ce775137b3366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006eaf3082450a908ce775137b3366" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44985 } } }
I20260620 07:04:39.788661 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:39.788724 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366 [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:39.795920 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:39.800514 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:39.808984 20948 catalog_manager.cc:1347] Generated new cluster ID: 67cc690960cc4148b813f73ff97cdc7c
I20260620 07:04:39.809161 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:39.824420 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:39.825812 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:39.840618 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd006eaf3082450a908ce775137b3366: Generated new TSK 0
I20260620 07:04:39.841581 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:04:39.851142 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--builtin_ntp_servers=127.11.81.212:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 07:04:40.183144 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:40.183467 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:40.183599 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:40.191038 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:40.191152 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:40.191320 20958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:40.200748 20958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:40.202142 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:40.203955 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:40.217249 20963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:40.217223 20964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:40.217432 20966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:40.218027 20958 server_base.cc:1034] running on GCE node
I20260620 07:04:40.218597 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:40.219576 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:40.220767 20958 hybrid_clock.cc:648] HybridClock initialized: now 1781939080220697 us; error 62 us; skew 500 ppm
I20260620 07:04:40.221067 20958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:40.223491 20958 webserver.cc:460] Webserver started at http://127.11.81.193:37203/ using document root <none> and password file <none>
I20260620 07:04:40.224190 20958 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:40.224275 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:40.224575 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:40.227502 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "43c80bae19bc48d98c6cc6ecc87e22f9"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "ed18b500dce73457d9d4edfafd1fa87f"
server_key_iv: "e088c73e159c41f6ef2633ffc1e44f97"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.228286 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "43c80bae19bc48d98c6cc6ecc87e22f9"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "ed18b500dce73457d9d4edfafd1fa87f"
server_key_iv: "e088c73e159c41f6ef2633ffc1e44f97"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.233582 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:40.237063 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.238487 20958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:40.238655 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "43c80bae19bc48d98c6cc6ecc87e22f9"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "ed18b500dce73457d9d4edfafd1fa87f"
server_key_iv: "e088c73e159c41f6ef2633ffc1e44f97"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.238869 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:40.260424 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:40.261337 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:40.261654 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:40.262629 20958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:40.264107 20958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:40.264209 20958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.264314 20958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:40.264382 20958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.294037 20958 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:37703
I20260620 07:04:40.294107 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:37703 every 8 connection(s)
I20260620 07:04:40.295456 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:40.303057 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 20958
I20260620 07:04:40.303242 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:40.303898 11591 external_mini_cluster.cc:1428] Setting key c7329f2af6cd1e7df3fec7d0d7358255
I20260620 07:04:40.306854 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--builtin_ntp_servers=127.11.81.212:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:04:40.308622 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:40.297950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46847 (local address 127.11.81.254:44985)
0620 07:04:40.298365 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:40.298370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:40.299620 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 07:04:40.301433 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:40.301438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:40.301528 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:40.301641 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:40.302809 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.303643 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.305760 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.306221 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.306342 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:40.307815 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:40.307820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:40.307823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:40.307841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:40.307883 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:40.307894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:40.307969 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:40.308191 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 07:04:40.308291 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":191,"threads_started":1}
I20260620 07:04:40.309665 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:40.298542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44985 (local address 127.11.81.193:46847)
0620 07:04:40.299464 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:40.299501 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:40.301144 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:40.301909 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:40.301918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:40.302016 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:40.302616 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:40.302641 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.303830 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.303836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:40.305599 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:40.305614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.306479 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.306488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:40.306736 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:40.307481 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:40.307484 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:40.307527 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:40.307632 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:40.307981 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:40.307993 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:40.308001 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:04:40.308167 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 07:04:40.308596 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":259,"threads_started":1}
I20260620 07:04:40.311234 21085 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44985
I20260620 07:04:40.311573 21085 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:40.312330 21085 heartbeater.cc:511] Master 127.11.81.254:44985 requested a full tablet report, sending...
I20260620 07:04:40.314383 20899 ts_manager.cc:194] Registered new tserver with Master: 43c80bae19bc48d98c6cc6ecc87e22f9 (127.11.81.193:37703)
I20260620 07:04:40.315932 20899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.193:46847
W20260620 07:04:40.624167 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:40.624471 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:40.624596 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:40.631966 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:40.632084 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:40.632230 21089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:40.641438 21089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:40.642836 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:40.644662 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:40.657692 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:40.657802 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:40.657778 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:40.658478 21089 server_base.cc:1034] running on GCE node
I20260620 07:04:40.659135 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:40.660125 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:40.661331 21089 hybrid_clock.cc:648] HybridClock initialized: now 1781939080661270 us; error 56 us; skew 500 ppm
I20260620 07:04:40.661650 21089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:40.663990 21089 webserver.cc:460] Webserver started at http://127.11.81.194:33379/ using document root <none> and password file <none>
I20260620 07:04:40.664705 21089 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:40.664806 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:40.665112 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:40.668051 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "0ddc6ef6572e45e2a52da08298c0ebf0"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "9de369e93b706ad57e65a15af1c2d0ae"
server_key_iv: "863757e43294ca428d3fa47a9de803e4"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.668870 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "0ddc6ef6572e45e2a52da08298c0ebf0"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "9de369e93b706ad57e65a15af1c2d0ae"
server_key_iv: "863757e43294ca428d3fa47a9de803e4"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.674284 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:04:40.678198 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.679697 21089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:40.679872 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "0ddc6ef6572e45e2a52da08298c0ebf0"
format_stamp: "Formatted at 2026-06-20 07:04:40 on dist-test-slave-wvz3"
server_key: "9de369e93b706ad57e65a15af1c2d0ae"
server_key_iv: "863757e43294ca428d3fa47a9de803e4"
server_key_version: "encryptionkey@0"
I20260620 07:04:40.680087 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:40.713339 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:40.714226 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:40.714504 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:40.715523 21089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:40.717015 21089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:40.717110 21089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.717222 21089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:40.717293 21089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:40.748261 21089 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:40761
I20260620 07:04:40.748314 21215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:40761 every 8 connection(s)
I20260620 07:04:40.749612 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:40.759630 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21089
I20260620 07:04:40.759783 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:40.760197 11591 external_mini_cluster.cc:1428] Setting key b7c943c3115a40ff544f8b70dbe8fa84
I20260620 07:04:40.763147 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--builtin_ntp_servers=127.11.81.212:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:04:40.764305 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:40.752386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:38407 (local address 127.11.81.254:44985)
0620 07:04:40.752589 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:40.752595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:40.754147 (+ 1552us) server_negotiation.cc:407] Connection header received
0620 07:04:40.756749 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:40.756753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:40.756821 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:40.756908 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:40.758285 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.759028 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.761428 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.761975 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.762097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:40.763525 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:40.763529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:40.763533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:40.763549 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:40.763581 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:40.763592 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:40.763669 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:40.763885 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:04:40.763984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:04:40.765343 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:40.752913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44985 (local address 127.11.81.194:38407)
0620 07:04:40.753956 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:40.754017 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:40.756436 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:40.757184 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:40.757196 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:40.757315 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:40.758095 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:40.758120 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.759180 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.759186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:40.761256 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:40.761272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:40.762184 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:40.762192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:40.762417 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:40.763061 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:40.763064 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:40.763103 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:40.763185 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:40.763688 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:40.763699 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:40.763707 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:04:40.763861 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:04:40.764311 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
I20260620 07:04:40.766746 21216 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44985
I20260620 07:04:40.767082 21216 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:40.768081 21216 heartbeater.cc:511] Master 127.11.81.254:44985 requested a full tablet report, sending...
I20260620 07:04:40.769721 20899 ts_manager.cc:194] Registered new tserver with Master: 0ddc6ef6572e45e2a52da08298c0ebf0 (127.11.81.194:40761)
I20260620 07:04:40.770648 20899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.194:38407
W20260620 07:04:41.080643 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:41.080927 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:41.081032 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:41.088514 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:41.088617 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:41.088758 21219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:41.098035 21219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:41.099303 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:41.101045 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:41.113646 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:41.113683 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:41.114172 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:41.114480 21219 server_base.cc:1034] running on GCE node
I20260620 07:04:41.115072 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:41.116077 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:41.117292 21219 hybrid_clock.cc:648] HybridClock initialized: now 1781939081117229 us; error 48 us; skew 500 ppm
I20260620 07:04:41.117627 21219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:41.119940 21219 webserver.cc:460] Webserver started at http://127.11.81.195:39791/ using document root <none> and password file <none>
I20260620 07:04:41.120640 21219 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:41.120751 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:41.121058 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:41.123991 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "da2d19e5b31e4ed98d832e252c69e08b"
format_stamp: "Formatted at 2026-06-20 07:04:41 on dist-test-slave-wvz3"
server_key: "b9b93193022048b58c2162d8a48f36db"
server_key_iv: "5a34167368732bf648e5b470b8194955"
server_key_version: "encryptionkey@0"
I20260620 07:04:41.124804 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "da2d19e5b31e4ed98d832e252c69e08b"
format_stamp: "Formatted at 2026-06-20 07:04:41 on dist-test-slave-wvz3"
server_key: "b9b93193022048b58c2162d8a48f36db"
server_key_iv: "5a34167368732bf648e5b470b8194955"
server_key_version: "encryptionkey@0"
I20260620 07:04:41.130249 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:04:41.134159 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:41.135758 21219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:41.135936 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "da2d19e5b31e4ed98d832e252c69e08b"
format_stamp: "Formatted at 2026-06-20 07:04:41 on dist-test-slave-wvz3"
server_key: "b9b93193022048b58c2162d8a48f36db"
server_key_iv: "5a34167368732bf648e5b470b8194955"
server_key_version: "encryptionkey@0"
I20260620 07:04:41.136154 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:41.161273 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:41.162189 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:41.162477 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:41.163540 21219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:41.165038 21219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:41.165133 21219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:41.165239 21219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:41.165310 21219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:41.195549 21219 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36279
I20260620 07:04:41.195653 21345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36279 every 8 connection(s)
I20260620 07:04:41.196861 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:41.205940 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21219
I20260620 07:04:41.206105 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:41.206552 11591 external_mini_cluster.cc:1428] Setting key 93931bb9280a629fa60b48f28ea51cf1
I20260620 07:04:41.209648 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:41.199484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:39449 (local address 127.11.81.254:44985)
0620 07:04:41.199689 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:41.199695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:41.201104 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 07:04:41.202904 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:41.202909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:41.202980 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:41.203070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:41.204243 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.204954 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.206716 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.207332 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.207461 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:41.208826 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:41.208831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:41.208835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:41.208851 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:41.208888 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:41.208899 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:41.208974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:41.209253 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 07:04:41.209336 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:04:41.210758 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:41.200029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44985 (local address 127.11.81.195:39449)
0620 07:04:41.200952 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:41.200989 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:41.202638 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:41.203367 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:41.203376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:41.203480 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:41.204071 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:41.204088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.205078 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.205082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:41.206558 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:41.206570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.207561 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.207568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:41.207802 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:41.208509 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:41.208512 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:41.208567 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:41.208655 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:41.208990 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:41.209000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:41.209008 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:04:41.209216 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 07:04:41.209643 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260620 07:04:41.212273 21346 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44985
I20260620 07:04:41.212534 21346 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:41.213269 21346 heartbeater.cc:511] Master 127.11.81.254:44985 requested a full tablet report, sending...
I20260620 07:04:41.214607 20899 ts_manager.cc:194] Registered new tserver with Master: da2d19e5b31e4ed98d832e252c69e08b (127.11.81.195:36279)
I20260620 07:04:41.215469 20899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.195:39449
I20260620 07:04:41.222096 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:04:41.228094 21087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:41.226547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.11.81.254:44985)
0620 07:04:41.226788 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:41.226795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:41.227159 (+ 364us) server_negotiation.cc:407] Connection header received
0620 07:04:41.227306 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:41.227312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:41.227407 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:41.227495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:41.227842 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49300: BlockingRecv error: recv got EOF from 127.0.0.1:49300 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 07:04:41.228812 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20958
I20260620 07:04:41.236951 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:41.229819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.11.81.193:37703)
0620 07:04:41.230371 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:41.230385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:41.230410 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:04:41.230461 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:41.230467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:41.230679 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:41.230814 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:41.231245 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.232248 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.233510 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.234076 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.234200 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:41.235237 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:41.235246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:41.235260 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:41.235974 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:41.236030 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:41.236051 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:41.236140 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:41.236462 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 07:04:41.236601 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260620 07:04:41.238222 21080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:41.318934 21085 heartbeater.cc:503] Master 127.11.81.254:44985 was elected leader, sending a full tablet report...
I20260620 07:04:41.319753 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20958
W20260620 07:04:41.333328 20884 connection.cc:535] server connection from 127.11.81.193:46847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:04:41.333577 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21089
I20260620 07:04:41.345791 21357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:41.334429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.11.81.194:40761)
0620 07:04:41.334949 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:41.334956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:41.334984 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:04:41.335050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:41.335056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:41.335358 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:41.335528 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:41.335910 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.336920 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.338112 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.338679 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.338788 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:41.344320 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:41.344329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:41.344344 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:41.344960 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:41.345022 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:41.345044 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:41.345194 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:41.345310 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:04:41.345457 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260620 07:04:41.347007 21211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:41.427644 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21089
I20260620 07:04:41.441084 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21219
I20260620 07:04:41.448987 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:41.441944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.11.81.195:36279)
0620 07:04:41.442579 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:41.442585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:41.442613 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:04:41.442672 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:41.442679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:41.442890 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:41.443027 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:41.443512 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.444591 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.445839 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:41.446397 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:41.446509 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:41.447588 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:41.447597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:41.447611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:41.448187 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:41.448236 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:41.448255 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:41.448371 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:41.448505 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 07:04:41.448636 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":341,"threads_started":1}
I20260620 07:04:41.450240 21341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:41.535987 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21219
I20260620 07:04:41.548662 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 20868
I20260620 07:04:41.549849 20929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:41.652216 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 20868
2026-06-20T07:04:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2401 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T07:04:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:41Z Disabled control of system clock
I20260620 07:04:41.719547 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39845
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:04:42.040763 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:42.041077 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:42.041174 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:42.048648 21365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:42.048774 21365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:42.048828 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:42.048918 21365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:42.048975 21365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:42.058311 21365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:42.059682 21365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:42.061440 21365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:42.073318 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:42.073285 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:42.073808 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:42.074404 21365 server_base.cc:1034] running on GCE node
I20260620 07:04:42.075205 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:42.076732 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:42.077987 21365 hybrid_clock.cc:648] HybridClock initialized: now 1781939082077941 us; error 32 us; skew 500 ppm
I20260620 07:04:42.078394 21365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:42.080806 21365 webserver.cc:460] Webserver started at http://127.11.81.254:38773/ using document root <none> and password file <none>
I20260620 07:04:42.081590 21365 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:42.081681 21365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:42.081995 21365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:42.085014 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "1770d1b52f574286bb568ed2d71ea3ec"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "94bcc2d4bfb7d8aa0279c049267c3be6"
server_key_iv: "120973410e081157e732b8cec34255bc"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.085831 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "1770d1b52f574286bb568ed2d71ea3ec"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "94bcc2d4bfb7d8aa0279c049267c3be6"
server_key_iv: "120973410e081157e732b8cec34255bc"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.091104 21365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:04:42.094548 21379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:42.095917 21365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:42.096086 21365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "1770d1b52f574286bb568ed2d71ea3ec"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "94bcc2d4bfb7d8aa0279c049267c3be6"
server_key_iv: "120973410e081157e732b8cec34255bc"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.096289 21365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:42.101903 21365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:42.102646 21365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:42.102911 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:42.120028 21365 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:39845
I20260620 07:04:42.120105 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:39845 every 8 connection(s)
I20260620 07:04:42.121387 21365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:42.125130 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:42.130573 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21365
I20260620 07:04:42.130764 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:42.131386 11591 external_mini_cluster.cc:1428] Setting key be96e8fe959df2802853ea630c5611cc
I20260620 07:04:42.135673 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec: Bootstrap starting.
I20260620 07:04:42.139104 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:42.139705 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:42.133447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.11.81.254:39845)
0620 07:04:42.134416 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:42.134427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:42.134465 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:04:42.136137 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:42.136160 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:42.136582 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:42.136909 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:42.138014 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:42.138021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:42.138034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:42.138050 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:42.138121 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:42.138138 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:42.138322 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:42.138424 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:04:42.138683 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":323,"threads_started":1}
I20260620 07:04:42.140199 21431 log.cc:824] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:42.143272 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec: No bootstrap required, opened a new log
I20260620 07:04:42.150645 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } }
I20260620 07:04:42.151041 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:42.151130 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1770d1b52f574286bb568ed2d71ea3ec, State: Initialized, Role: FOLLOWER
I20260620 07:04:42.151716 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } }
I20260620 07:04:42.151923 21431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:42.152117 21431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:42.152271 21431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:42.154470 21431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } }
I20260620 07:04:42.155041 21431 leader_election.cc:302] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1770d1b52f574286bb568ed2d71ea3ec; no voters:
I20260620 07:04:42.155753 21431 leader_election.cc:288] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:42.155859 21436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:42.156549 21436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [term 1 LEADER]: Becoming Leader. State: Replica: 1770d1b52f574286bb568ed2d71ea3ec, State: Running, Role: LEADER
I20260620 07:04:42.157088 21436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } }
I20260620 07:04:42.157720 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:42.159716 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1770d1b52f574286bb568ed2d71ea3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } } }
I20260620 07:04:42.159691 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1770d1b52f574286bb568ed2d71ea3ec. Latest consensus state: current_term: 1 leader_uuid: "1770d1b52f574286bb568ed2d71ea3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1770d1b52f574286bb568ed2d71ea3ec" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39845 } } }
I20260620 07:04:42.160202 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:42.160200 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:42.162251 21443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:42.167719 21443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:42.176930 21443 catalog_manager.cc:1347] Generated new cluster ID: c139fda3aa11403e9cdfa2cafae29a5d
I20260620 07:04:42.177121 21443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:42.204115 21443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:42.204555 21443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:42.211606 21443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1770d1b52f574286bb568ed2d71ea3ec: Generated new TSK 0
I20260620 07:04:42.212586 21443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:04:42.226817 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--builtin_ntp_servers=127.11.81.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:04:42.573621 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:42.573916 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:42.574014 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:42.581444 21455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:42.581555 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:42.581665 21455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:42.590812 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:42.592105 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:42.593792 21455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:42.606448 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:42.606616 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:42.606638 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:42.607270 21455 server_base.cc:1034] running on GCE node
I20260620 07:04:42.607898 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:42.608879 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:42.610126 21455 hybrid_clock.cc:648] HybridClock initialized: now 1781939082610074 us; error 45 us; skew 500 ppm
I20260620 07:04:42.610440 21455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:42.612731 21455 webserver.cc:460] Webserver started at http://127.11.81.193:37063/ using document root <none> and password file <none>
I20260620 07:04:42.613427 21455 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:42.613518 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:42.613833 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:42.618228 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "1e627bc4fb4845c88b677fb48bcc2f48"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "d539bf3aad793fd4ec76bc4e9d20c47b"
server_key_iv: "e2fdb0f4b8373c555298b1677d681376"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.619036 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "1e627bc4fb4845c88b677fb48bcc2f48"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "d539bf3aad793fd4ec76bc4e9d20c47b"
server_key_iv: "e2fdb0f4b8373c555298b1677d681376"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.624334 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 07:04:42.627705 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:42.629045 21455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:04:42.629191 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "1e627bc4fb4845c88b677fb48bcc2f48"
format_stamp: "Formatted at 2026-06-20 07:04:42 on dist-test-slave-wvz3"
server_key: "d539bf3aad793fd4ec76bc4e9d20c47b"
server_key_iv: "e2fdb0f4b8373c555298b1677d681376"
server_key_version: "encryptionkey@0"
I20260620 07:04:42.629398 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:42.634909 21455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:42.635705 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:42.635944 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:42.636791 21455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:42.638139 21455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:42.638229 21455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:42.638335 21455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:42.638401 21455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:42.668218 21455 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:35595
I20260620 07:04:42.668263 21581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:35595 every 8 connection(s)
I20260620 07:04:42.669564 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:42.672078 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21455
I20260620 07:04:42.672325 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:42.673151 11591 external_mini_cluster.cc:1428] Setting key ff139510875315fec65c9664b70aee51
I20260620 07:04:42.676385 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--builtin_ntp_servers=127.11.81.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:04:42.678118 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:42.672090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:40937 (local address 127.11.81.254:39845)
0620 07:04:42.674311 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:42.674317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:42.674334 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:04:42.675822 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:42.675828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:42.675889 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:42.676004 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:42.677354 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:42.677360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:42.677364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:42.677385 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:42.677437 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:42.677453 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:42.677573 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:42.677782 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:04:42.677842 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2112,"thread_start_us":2017,"threads_started":1}
I20260620 07:04:42.679075 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:42.672653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39845 (local address 127.11.81.193:40937)
0620 07:04:42.673692 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:42.673737 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:42.675567 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:42.676253 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:42.676262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:42.676360 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:42.677045 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:42.677047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:42.677087 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:42.677175 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:42.677578 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:42.677587 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:42.677594 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:04:42.677849 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 07:04:42.678103 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":319,"threads_started":1}
I20260620 07:04:42.680490 21582 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39845
I20260620 07:04:42.680806 21582 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:42.681499 21582 heartbeater.cc:511] Master 127.11.81.254:39845 requested a full tablet report, sending...
I20260620 07:04:42.683884 21396 ts_manager.cc:194] Registered new tserver with Master: 1e627bc4fb4845c88b677fb48bcc2f48 (127.11.81.193:35595)
W20260620 07:04:42.996706 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:42.997010 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:42.997128 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:43.004673 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:43.004799 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:43.004941 21586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:43.014196 21586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:43.015537 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:43.017349 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:43.030524 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:43.030499 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:43.030941 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:43.031579 21586 server_base.cc:1034] running on GCE node
I20260620 07:04:43.032231 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:43.033200 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:43.034437 21586 hybrid_clock.cc:648] HybridClock initialized: now 1781939083034386 us; error 35 us; skew 500 ppm
I20260620 07:04:43.034823 21586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:43.037382 21586 webserver.cc:460] Webserver started at http://127.11.81.194:40547/ using document root <none> and password file <none>
I20260620 07:04:43.038112 21586 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:43.038226 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:43.038540 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:43.041534 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "124cca3b87af43fdabb7ff3abb3e9135"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "c395eb47805ce9b4b04238603e94bdc6"
server_key_iv: "a8e01040a0145e11b7f994277a82d5f3"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.042316 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "124cca3b87af43fdabb7ff3abb3e9135"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "c395eb47805ce9b4b04238603e94bdc6"
server_key_iv: "a8e01040a0145e11b7f994277a82d5f3"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.047570 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:04:43.051193 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.052724 21586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:04:43.052896 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "124cca3b87af43fdabb7ff3abb3e9135"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "c395eb47805ce9b4b04238603e94bdc6"
server_key_iv: "a8e01040a0145e11b7f994277a82d5f3"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.053130 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:43.058920 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:43.059830 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:43.060081 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:43.060997 21586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:43.062492 21586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:43.062587 21586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.062692 21586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:43.062765 21586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.093183 21586 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:46383
I20260620 07:04:43.093240 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:46383 every 8 connection(s)
I20260620 07:04:43.094467 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:43.100216 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21586
I20260620 07:04:43.100391 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:43.100889 11591 external_mini_cluster.cc:1428] Setting key e9bfc16daa76c39e9a68124a14be97ec
I20260620 07:04:43.102754 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.097000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46591 (local address 127.11.81.254:39845)
0620 07:04:43.097168 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.097173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.098580 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 07:04:43.100426 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.100432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.100528 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.100645 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.102078 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:43.102083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:43.102086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:43.102102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:43.102134 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:43.102146 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:43.102252 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:43.102435 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 07:04:43.102499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:04:43.103622 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--builtin_ntp_servers=127.11.81.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:04:43.103808 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.097529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39845 (local address 127.11.81.194:46591)
0620 07:04:43.098406 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:43.098462 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:43.100110 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:43.100837 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:43.100846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:43.100965 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:43.101708 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:43.101712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:43.101765 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:43.101882 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:43.102235 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:43.102245 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:43.102253 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:04:43.102521 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 07:04:43.102790 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":257,"threads_started":1}
I20260620 07:04:43.105192 21713 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39845
I20260620 07:04:43.105473 21713 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:43.106019 21713 heartbeater.cc:511] Master 127.11.81.254:39845 requested a full tablet report, sending...
I20260620 07:04:43.107442 21396 ts_manager.cc:194] Registered new tserver with Master: 124cca3b87af43fdabb7ff3abb3e9135 (127.11.81.194:46383)
W20260620 07:04:43.429283 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:43.429555 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:43.429670 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:43.437080 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:43.437183 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:43.437328 21716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:43.446913 21716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:43.448252 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:43.450049 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:43.462663 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:43.462852 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:43.462742 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:43.463361 21716 server_base.cc:1034] running on GCE node
I20260620 07:04:43.464123 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:43.465059 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:43.466290 21716 hybrid_clock.cc:648] HybridClock initialized: now 1781939083466216 us; error 64 us; skew 500 ppm
I20260620 07:04:43.466643 21716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:04:43.468922 21716 webserver.cc:460] Webserver started at http://127.11.81.195:33441/ using document root <none> and password file <none>
I20260620 07:04:43.469620 21716 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:43.469715 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:43.470014 21716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:43.472874 21716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "17cdfd3365b841f39a727d0117dd1abb"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "24a1963bf8bc881e729eb9bb919f4019"
server_key_iv: "da84d11852c8ed66ab4cd36a52c5df01"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.473651 21716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "17cdfd3365b841f39a727d0117dd1abb"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "24a1963bf8bc881e729eb9bb919f4019"
server_key_iv: "da84d11852c8ed66ab4cd36a52c5df01"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.479233 21716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:04:43.483318 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.485222 21716 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:04:43.485394 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "17cdfd3365b841f39a727d0117dd1abb"
format_stamp: "Formatted at 2026-06-20 07:04:43 on dist-test-slave-wvz3"
server_key: "24a1963bf8bc881e729eb9bb919f4019"
server_key_iv: "da84d11852c8ed66ab4cd36a52c5df01"
server_key_version: "encryptionkey@0"
I20260620 07:04:43.485639 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:43.491338 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:43.492125 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:43.492367 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:43.493248 21716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:43.494629 21716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:43.494715 21716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.494819 21716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:43.494881 21716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:43.523837 21716 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33277
I20260620 07:04:43.523885 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33277 every 8 connection(s)
I20260620 07:04:43.525146 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:43.526037 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21716
I20260620 07:04:43.526211 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:43.526661 11591 external_mini_cluster.cc:1428] Setting key 0e8bbc11d296a23458b49391bbb56a33
I20260620 07:04:43.533398 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.527781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:43345 (local address 127.11.81.254:39845)
0620 07:04:43.527962 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.527968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.529791 (+ 1823us) server_negotiation.cc:407] Connection header received
0620 07:04:43.531502 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.531506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.531564 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.531637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.532848 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:43.532853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:43.532856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:43.532873 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:43.532906 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:43.532917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:43.533022 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:43.533155 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 07:04:43.533199 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:04:43.534381 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.528511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39845 (local address 127.11.81.195:43345)
0620 07:04:43.529640 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:43.529694 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:43.531216 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:43.531821 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:43.531829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:43.531919 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:43.532558 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:04:43.532561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:43.532610 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:04:43.532694 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:43.532999 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:43.533005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:43.533010 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:04:43.533220 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 07:04:43.533415 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":310,"threads_started":1}
I20260620 07:04:43.535772 21843 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39845
I20260620 07:04:43.536046 21843 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:43.536612 21843 heartbeater.cc:511] Master 127.11.81.254:39845 requested a full tablet report, sending...
I20260620 07:04:43.537900 21396 ts_manager.cc:194] Registered new tserver with Master: 17cdfd3365b841f39a727d0117dd1abb (127.11.81.195:33277)
I20260620 07:04:43.541963 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:04:43.547845 21584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:04:43.546418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.11.81.254:39845)
0620 07:04:43.546622 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.546629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.547000 (+ 371us) server_negotiation.cc:407] Connection header received
0620 07:04:43.547184 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.547188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.547238 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.547345 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.547592 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57734: BlockingRecv error: recv got EOF from 127.0.0.1:57734 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:04:43.548363 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21455
I20260620 07:04:43.551857 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.549322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.11.81.193:35595)
0620 07:04:43.549754 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.549761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.549787 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:04:43.549841 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.549846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.550044 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.550172 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.551166 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:43.551172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:43.551184 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:43.551197 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:43.551246 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:43.551263 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:43.551400 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:43.551501 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 07:04:43.551588 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":193,"threads_started":1}
I20260620 07:04:43.553038 21577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:43.632962 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21455
I20260620 07:04:43.646370 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21586
I20260620 07:04:43.650202 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.647367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.11.81.194:46383)
0620 07:04:43.647830 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.647837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.647862 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:04:43.647926 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.647934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.648300 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.648468 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.649482 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:43.649489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:43.649500 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:43.649518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:43.649575 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:43.649590 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:43.649709 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:43.649836 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 07:04:43.649947 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":213,"threads_started":1}
I20260620 07:04:43.651597 21708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:43.731457 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21586
I20260620 07:04:43.744040 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21716
I20260620 07:04:43.748032 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:43.745079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.11.81.195:33277)
0620 07:04:43.745696 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:43.745702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:43.745732 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:04:43.745793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:43.745798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:43.746012 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:43.746170 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:43.747267 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:43.747273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:43.747324 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:04:43.747346 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:43.747388 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:04:43.747404 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:43.747577 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:43.747650 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:04:43.747748 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":305,"threads_started":1}
I20260620 07:04:43.749194 21838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:43.828240 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21716
I20260620 07:04:43.840817 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21365
I20260620 07:04:43.842016 21426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:43.942245 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21365
2026-06-20T07:04:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2291 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:44 dist-test-slave-wvz3 krb5kdc[21861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:44 dist-test-slave-wvz3 krb5kdc[21861](info): set up 2 sockets
Jun 20 07:04:44 dist-test-slave-wvz3 krb5kdc[21861](info): commencing operation
krb5kdc: starting...
W20260620 07:04:44.960021 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:45 dist-test-slave-wvz3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:04:45.179737 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40297
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:45.504308 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:45.504631 21874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:45.504770 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:45.512207 21874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:04:45.512341 21874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:45.512424 21874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:45.512501 21874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:04:45.512563 21874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:04:45.522868 21874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:45.524307 21874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:45.525573 21874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:45.536939 21879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:45.536978 21880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:45.536908 21882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:45.538430 21874 server_base.cc:1034] running on GCE node
I20260620 07:04:45.539263 21874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:45.540979 21874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:45.542260 21874 hybrid_clock.cc:648] HybridClock initialized: now 1781939085542196 us; error 56 us; skew 500 ppm
I20260620 07:04:45.542671 21874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:45 dist-test-slave-wvz3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:45.546051 21874 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:04:45.547741 21874 webserver.cc:460] Webserver started at http://127.11.81.254:33381/ using document root <none> and password file <none>
I20260620 07:04:45.548462 21874 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:45.548605 21874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:45.548941 21874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:45.552220 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "5b0ca39635a04feda2d0d5b8501b7abc"
format_stamp: "Formatted at 2026-06-20 07:04:45 on dist-test-slave-wvz3"
server_key: "42dadb3c97495054da8056e4fe3f1488"
server_key_iv: "b301bf0f7ab93d2f22e3f92377dac0d6"
server_key_version: "encryptionkey@0"
I20260620 07:04:45.553042 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "5b0ca39635a04feda2d0d5b8501b7abc"
format_stamp: "Formatted at 2026-06-20 07:04:45 on dist-test-slave-wvz3"
server_key: "42dadb3c97495054da8056e4fe3f1488"
server_key_iv: "b301bf0f7ab93d2f22e3f92377dac0d6"
server_key_version: "encryptionkey@0"
I20260620 07:04:45.558669 21874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:04:45.562417 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:45.563967 21874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:04:45.564146 21874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "5b0ca39635a04feda2d0d5b8501b7abc"
format_stamp: "Formatted at 2026-06-20 07:04:45 on dist-test-slave-wvz3"
server_key: "42dadb3c97495054da8056e4fe3f1488"
server_key_iv: "b301bf0f7ab93d2f22e3f92377dac0d6"
server_key_version: "encryptionkey@0"
I20260620 07:04:45.564340 21874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:45.584882 21874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:45.589267 21874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:45.589536 21874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:45.607115 21874 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:40297
I20260620 07:04:45.607157 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:40297 every 8 connection(s)
I20260620 07:04:45.608486 21874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:04:45.611742 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21874
I20260620 07:04:45.611994 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:04:45.612705 21941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:45.612795 11591 external_mini_cluster.cc:1428] Setting key 68f0f116bd637a7ef0aa7cced4153ea2
I20260620 07:04:45.627190 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: Bootstrap starting.
Jun 20 07:04:45 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:45.630734 21941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:45.631844 21941 log.cc:824] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:45.634644 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: No bootstrap required, opened a new log
I20260620 07:04:45.636073 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:45.614888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.11.81.254:40297)
0620 07:04:45.615737 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:45.615748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:45.615793 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:04:45.617498 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:45.617523 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:45.617961 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:45.618332 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:45.620153 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:45.621456 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:45.623241 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:45.624055 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:45.624214 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:45.628856 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:45.628865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:45.628881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:45.628926 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:45.632135 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:45.632865 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:45.632873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:45.632880 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:45.633055 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:45.633415 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:45.633420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:45.633424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:45.633946 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:45.634228 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:45.634634 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 07:04:45.635045 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":398,"threads_started":1}
I20260620 07:04:45.643399 21941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } }
I20260620 07:04:45.643797 21941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:45.643898 21941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0ca39635a04feda2d0d5b8501b7abc, State: Initialized, Role: FOLLOWER
I20260620 07:04:45.644456 21941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } }
I20260620 07:04:45.644764 21941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:04:45.644896 21941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:04:45.645048 21941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:45.647182 21941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } }
I20260620 07:04:45.647707 21941 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0ca39635a04feda2d0d5b8501b7abc; no voters:
I20260620 07:04:45.648306 21941 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:04:45.648463 21946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:45.649528 21946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [term 1 LEADER]: Becoming Leader. State: Replica: 5b0ca39635a04feda2d0d5b8501b7abc, State: Running, Role: LEADER
I20260620 07:04:45.650059 21946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } }
I20260620 07:04:45.650386 21941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:04:45.652887 21947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } } }
I20260620 07:04:45.652971 21948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0ca39635a04feda2d0d5b8501b7abc. Latest consensus state: current_term: 1 leader_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0ca39635a04feda2d0d5b8501b7abc" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40297 } } }
I20260620 07:04:45.653347 21948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:45.653249 21947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc [sys.catalog]: This master's current role is: LEADER
I20260620 07:04:45.656170 21953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:04:45.661331 21953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:04:45.669459 21953 catalog_manager.cc:1347] Generated new cluster ID: 2dad692a404141049b5841f24e1d4ff9
I20260620 07:04:45.669591 21953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:04:45.683624 21953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:04:45.685559 21953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:04:45.706085 21953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: Generated new TSK 0
I20260620 07:04:45.707198 21953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:04:45.769313 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:46.091831 21967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:46.092144 21967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:46.092270 21967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:46.100028 21967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:46.100203 21967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:46.100382 21967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:04:46.110610 21967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:46.112237 21967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:46.113571 21967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:46.126590 21973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:46.126688 21972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:46.126631 21975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:46.127395 21967 server_base.cc:1034] running on GCE node
I20260620 07:04:46.128069 21967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:46.129108 21967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:46.130311 21967 hybrid_clock.cc:648] HybridClock initialized: now 1781939086130257 us; error 49 us; skew 500 ppm
I20260620 07:04:46.130640 21967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:46 dist-test-slave-wvz3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939086, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:46.133769 21967 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:04:46.135272 21967 webserver.cc:460] Webserver started at http://127.11.81.193:34251/ using document root <none> and password file <none>
I20260620 07:04:46.135978 21967 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:46.136101 21967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:46.136381 21967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:46.139269 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "ed90d5b0778e615f85d6881af4da9583"
server_key_iv: "308d17b1beb62b855fd59a9165e25087"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.140065 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "ed90d5b0778e615f85d6881af4da9583"
server_key_iv: "308d17b1beb62b855fd59a9165e25087"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.145315 21967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 07:04:46.148860 21982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:46.150229 21967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:46.150363 21967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "ed90d5b0778e615f85d6881af4da9583"
server_key_iv: "308d17b1beb62b855fd59a9165e25087"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.150527 21967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:46.175262 21967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:46.179375 21967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:46.179630 21967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:46.180608 21967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:46.181929 21967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:46.182010 21967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:46.182094 21967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:46.182137 21967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:46.212868 21967 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:42803
I20260620 07:04:46.212965 22094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:42803 every 8 connection(s)
I20260620 07:04:46.214254 21967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:04:46.224082 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 21967
I20260620 07:04:46.224294 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:04:46.225066 11591 external_mini_cluster.cc:1428] Setting key c7baff9a5da44b75affca230def0bfa9
Jun 20 07:04:46 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939086, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:46.236011 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:46.216843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:40545 (local address 127.11.81.254:40297)
0620 07:04:46.217372 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:46.217378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:46.218348 (+ 970us) server_negotiation.cc:407] Connection header received
0620 07:04:46.220272 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:46.220277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:46.220356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:46.220462 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:46.222118 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.222825 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.224558 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.225166 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.225303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:46.229551 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:46.229558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:46.229562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:46.229603 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:46.232681 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:46.233537 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:46.233544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:46.233547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:46.233697 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:46.234113 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:46.234117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:46.234120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:46.234453 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:46.234608 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:46.235391 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 07:04:46.235634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260620 07:04:46.236918 22098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:46.217404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40297 (local address 127.11.81.193:40545)
0620 07:04:46.218179 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:46.218225 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:46.219980 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:46.220728 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:46.220737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:46.221316 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:46.221930 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:46.221948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.222970 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.222974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:46.224372 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:46.224386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.225390 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.225396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:46.225604 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:46.226796 (+ 1192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:46.226821 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:46.229289 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:46.232864 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:46.232873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:46.232889 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:46.233397 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:46.233793 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:46.233797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:46.233799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:46.233986 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:46.234589 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:46.234595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:46.234931 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 07:04:46.235192 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 07:04:46.235673 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":290,"threads_started":1}
I20260620 07:04:46.238432 22095 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40297
I20260620 07:04:46.238729 22095 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:46.239550 22095 heartbeater.cc:511] Master 127.11.81.254:40297 requested a full tablet report, sending...
I20260620 07:04:46.241636 21906 ts_manager.cc:194] Registered new tserver with Master: 3b72ef1ce0e5458fb38f1c6976c8b89f (127.11.81.193:42803)
I20260620 07:04:46.243069 21906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:40545
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:04:46.274744 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:46.597393 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:46.597679 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:46.597772 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:46.605062 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:46.605185 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:46.605295 22101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:04:46.615089 22101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:46.616386 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:46.617687 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:46.630650 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:46.630736 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:46.630759 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:46.631413 22101 server_base.cc:1034] running on GCE node
I20260620 07:04:46.632027 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:46.632968 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:46.634169 22101 hybrid_clock.cc:648] HybridClock initialized: now 1781939086634117 us; error 41 us; skew 500 ppm
I20260620 07:04:46.634481 22101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:46 dist-test-slave-wvz3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939086, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:46.641691 22101 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:04:46.643152 22101 webserver.cc:460] Webserver started at http://127.11.81.194:33037/ using document root <none> and password file <none>
I20260620 07:04:46.643901 22101 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:46.644007 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:46.644273 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:46.647158 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "7bb7cb0e516c4019ba7a3ce695882470"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "7f98258c1f5d80b9155e5a3442facdcc"
server_key_iv: "ad9cb53333c694e5959fe488661bd9e6"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.648006 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb7cb0e516c4019ba7a3ce695882470"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "7f98258c1f5d80b9155e5a3442facdcc"
server_key_iv: "ad9cb53333c694e5959fe488661bd9e6"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.653326 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:04:46.657181 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:46.658651 22101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:04:46.658813 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "7bb7cb0e516c4019ba7a3ce695882470"
format_stamp: "Formatted at 2026-06-20 07:04:46 on dist-test-slave-wvz3"
server_key: "7f98258c1f5d80b9155e5a3442facdcc"
server_key_iv: "ad9cb53333c694e5959fe488661bd9e6"
server_key_version: "encryptionkey@0"
I20260620 07:04:46.659009 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:46.679790 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:46.684587 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:46.684871 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:46.685860 22101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:46.687229 22101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:46.687343 22101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:04:46.687470 22101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:46.687525 22101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:46.718179 22101 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:34721
I20260620 07:04:46.718228 22228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:34721 every 8 connection(s)
I20260620 07:04:46.719434 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:04:46.727581 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 22101
I20260620 07:04:46.727766 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:04:46.728313 11591 external_mini_cluster.cc:1428] Setting key 55b20fa63577aa933f74701e68d0e7e6
Jun 20 07:04:46 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939086, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:46.739655 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:46.722060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:57687 (local address 127.11.81.254:40297)
0620 07:04:46.722251 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:46.722255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:46.723365 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 07:04:46.725270 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:46.725275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:46.725352 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:46.725445 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:46.726968 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.727761 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.729743 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.730346 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.730458 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:46.734072 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:46.734077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:46.734080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:46.734113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:46.736507 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:46.737335 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:46.737341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:46.737345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:46.737496 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:46.737894 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:46.737899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:46.737902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:46.738242 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:46.738382 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:46.739070 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 07:04:46.739336 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:04:46.740617 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:46.722481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40297 (local address 127.11.81.194:57687)
0620 07:04:46.723192 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:46.723236 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:46.724970 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:46.725700 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:46.725708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:46.726199 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:46.726771 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:46.726788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.727928 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.727934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:46.729574 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:46.729589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:46.730579 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:46.730588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:46.730804 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:46.731469 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:46.731500 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:46.733837 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:46.736639 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:46.736648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:46.736667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:46.737228 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:46.737610 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:46.737615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:46.737619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:46.737801 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:46.738346 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:46.738351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:46.738691 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 07:04:46.738922 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:04:46.739365 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":266,"threads_started":1}
I20260620 07:04:46.742162 22229 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40297
I20260620 07:04:46.742486 22229 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:46.743360 22229 heartbeater.cc:511] Master 127.11.81.254:40297 requested a full tablet report, sending...
I20260620 07:04:46.745023 21906 ts_manager.cc:194] Registered new tserver with Master: 7bb7cb0e516c4019ba7a3ce695882470 (127.11.81.194:34721)
I20260620 07:04:46.745954 21906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:57687
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:04:46.778478 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:04:47.100526 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:04:47.100805 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:04:47.100932 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:04:47.108273 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:04:47.108394 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:04:47.108541 22234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:04:47.118618 22234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:04:47.119987 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:04:47.121341 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:04:47.134728 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:47.134704 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:04:47.135002 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:04:47.135509 22234 server_base.cc:1034] running on GCE node
I20260620 07:04:47.136145 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:04:47.137152 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:04:47.138375 22234 hybrid_clock.cc:648] HybridClock initialized: now 1781939087138318 us; error 51 us; skew 500 ppm
I20260620 07:04:47.138697 22234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:04:47 dist-test-slave-wvz3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939087, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:04:47.141978 22234 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:04:47.143563 22234 webserver.cc:460] Webserver started at http://127.11.81.195:34835/ using document root <none> and password file <none>
I20260620 07:04:47.144258 22234 fs_manager.cc:359] Metadata directory not provided
I20260620 07:04:47.144358 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:04:47.144611 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:04:47.147681 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "59ecf55fee2e4743a156d16041682459"
format_stamp: "Formatted at 2026-06-20 07:04:47 on dist-test-slave-wvz3"
server_key: "efaf98f8970a2fe4a0b50c2f28cd84e8"
server_key_iv: "a66254128dde86826fbbf81928c79582"
server_key_version: "encryptionkey@0"
I20260620 07:04:47.148389 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "59ecf55fee2e4743a156d16041682459"
format_stamp: "Formatted at 2026-06-20 07:04:47 on dist-test-slave-wvz3"
server_key: "efaf98f8970a2fe4a0b50c2f28cd84e8"
server_key_iv: "a66254128dde86826fbbf81928c79582"
server_key_version: "encryptionkey@0"
I20260620 07:04:47.153720 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 07:04:47.157528 22249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:47.158988 22234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:04:47.159128 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "59ecf55fee2e4743a156d16041682459"
format_stamp: "Formatted at 2026-06-20 07:04:47 on dist-test-slave-wvz3"
server_key: "efaf98f8970a2fe4a0b50c2f28cd84e8"
server_key_iv: "a66254128dde86826fbbf81928c79582"
server_key_version: "encryptionkey@0"
I20260620 07:04:47.159363 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:04:47.189152 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:04:47.193667 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:04:47.193953 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:04:47.194928 22234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:04:47.196357 22234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:04:47.196430 22234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:47.196552 22234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:04:47.196610 22234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:04:47.227072 22234 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:37569
I20260620 07:04:47.227123 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:37569 every 8 connection(s)
I20260620 07:04:47.228353 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:04:47.232733 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 22234
I20260620 07:04:47.232947 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:04:47.233389 11591 external_mini_cluster.cc:1428] Setting key c585b2d2bd2005ce8a9f260502e7aec2
Jun 20 07:04:47 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939087, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:04:47.246135 22095 heartbeater.cc:503] Master 127.11.81.254:40297 was elected leader, sending a full tablet report...
I20260620 07:04:47.248584 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.230995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:40123 (local address 127.11.81.254:40297)
0620 07:04:47.231164 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.231168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.232503 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 07:04:47.234435 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.234441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.234528 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.234634 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:47.236482 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.237238 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.238749 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.239235 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.239416 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.242576 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:47.242581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:47.242585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:47.242619 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:47.245192 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.245958 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.245963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.245966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.246117 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.246625 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.246632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.246635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.246963 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:47.247137 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.248007 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.248236 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:04:47.249611 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.231609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40297 (local address 127.11.81.195:40123)
0620 07:04:47.232331 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.232376 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.234101 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.234895 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.234907 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.235658 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:04:47.236303 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.236324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.237366 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.237370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.238597 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.238608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.239488 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.239492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.239689 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.240264 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:04:47.240285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:04:47.242383 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:04:47.245333 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:47.245341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:47.245356 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:47.245774 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:47.246234 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:04:47.246239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:04:47.246242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:04:47.246491 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:04:47.247121 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:04:47.247128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:04:47.247585 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.247896 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.248457 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":299,"threads_started":1}
I20260620 07:04:47.250955 22362 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40297
I20260620 07:04:47.251219 22362 heartbeater.cc:464] Registering TS with master...
I20260620 07:04:47.252012 22362 heartbeater.cc:511] Master 127.11.81.254:40297 requested a full tablet report, sending...
I20260620 07:04:47.253396 21906 ts_manager.cc:194] Registered new tserver with Master: 59ecf55fee2e4743a156d16041682459 (127.11.81.195:37569)
I20260620 07:04:47.254225 21906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:40123
I20260620 07:04:47.261204 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:04:47.277601 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.265697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.11.81.254:40297)
0620 07:04:47.265862 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.265866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.266343 (+ 477us) server_negotiation.cc:407] Connection header received
0620 07:04:47.266465 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.266470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.266543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.266648 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:47.268190 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.268990 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.270157 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.270584 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.270673 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.272785 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:47.272790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:47.272794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:47.272824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:47.275122 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.275719 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.275723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.275725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.275850 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.276298 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.276302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.276304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.276637 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:47.276886 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.277183 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.277370 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:04:47.281805 21906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:04:47.288719 21906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:04:47.322868 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.300594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:42803 (local address 127.0.0.1:52952)
0620 07:04:47.313927 (+ 13333us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.313946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.314078 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.315413 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.315422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.315458 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:47.316082 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.316095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.317882 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.317888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.319932 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.319944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.321857 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.321864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.322048 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.322097 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.322218 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.322415 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13201,"mutex_wait_us":173,"thread_start_us":12815,"threads_started":1}
I20260620 07:04:47.323055 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.301456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:37569 (local address 127.0.0.1:60718)
0620 07:04:47.314758 (+ 13302us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.314776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.314888 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.315616 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.315621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.315647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:47.316085 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.316095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.317879 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.317884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.320382 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.320402 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.322277 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.322284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.322480 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.322511 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.322596 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.322695 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13172,"mutex_wait_us":59,"thread_start_us":13063,"threads_started":1}
I20260620 07:04:47.323698 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.302208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.11.81.195:37569)
0620 07:04:47.314616 (+ 12408us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.314624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.315008 (+ 384us) server_negotiation.cc:407] Connection header received
0620 07:04:47.315114 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.315121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.315363 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.315534 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:47.316301 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.317689 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.320550 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.321902 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.322017 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.322226 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.323279 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.323459 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12261,"thread_start_us":12088,"threads_started":1}
I20260620 07:04:47.323796 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.301056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.11.81.193:42803)
0620 07:04:47.313277 (+ 12221us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.313288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.314183 (+ 895us) server_negotiation.cc:407] Connection header received
0620 07:04:47.314307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.314315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.315095 (+ 780us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.315332 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:47.316274 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.317721 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.320081 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.321341 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.321485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.321715 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.323373 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.323514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12012,"thread_start_us":12235,"threads_started":1}
I20260620 07:04:47.324213 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.300383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:34721 (local address 127.0.0.1:51812)
0620 07:04:47.312842 (+ 12459us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.312904 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.313179 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.316605 (+ 3426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.316610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.316637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:47.316966 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.316979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.318730 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.318735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.320768 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.320778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.322443 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.322449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.322615 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.322647 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.323100 (+ 453us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.323974 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12089,"mutex_wait_us":1,"thread_start_us":13251,"threads_started":1}
I20260620 07:04:47.324481 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.300601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.11.81.194:34721)
0620 07:04:47.316070 (+ 15469us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.316076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.316098 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:04:47.316168 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.316173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.316404 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.316600 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:47.317131 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.318586 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.320900 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.322229 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.322360 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.322541 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.323322 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.323485 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15317,"thread_start_us":15190,"threads_started":1}
I20260620 07:04:47.327384 22030 tablet_service.cc:1467] Processing CreateTablet for tablet ce50de71acb64c729db4b7f7765fa426 (DEFAULT_TABLE table=test-table [id=9a67982cbd404a54a3aa468fff267376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:47.327536 22297 tablet_service.cc:1467] Processing CreateTablet for tablet ce50de71acb64c729db4b7f7765fa426 (DEFAULT_TABLE table=test-table [id=9a67982cbd404a54a3aa468fff267376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:47.327709 22164 tablet_service.cc:1467] Processing CreateTablet for tablet ce50de71acb64c729db4b7f7765fa426 (DEFAULT_TABLE table=test-table [id=9a67982cbd404a54a3aa468fff267376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:04:47.328759 22030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce50de71acb64c729db4b7f7765fa426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:47.328915 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce50de71acb64c729db4b7f7765fa426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:47.328923 22297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce50de71acb64c729db4b7f7765fa426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:04:47.339972 22380 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Bootstrap starting.
I20260620 07:04:47.342919 22382 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Bootstrap starting.
I20260620 07:04:47.343713 22381 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Bootstrap starting.
I20260620 07:04:47.344959 22380 tablet_bootstrap.cc:654] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:47.346386 22380 log.cc:824] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:47.346771 22382 tablet_bootstrap.cc:654] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:47.347550 22381 tablet_bootstrap.cc:654] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Neither blocks nor log segments found. Creating new log.
I20260620 07:04:47.347826 22382 log.cc:824] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:47.348766 22381 log.cc:824] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Log is configured to *not* fsync() on all Append() calls
I20260620 07:04:47.350831 22380 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: No bootstrap required, opened a new log
I20260620 07:04:47.350898 22382 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: No bootstrap required, opened a new log
I20260620 07:04:47.351106 22380 ts_tablet_manager.cc:1397] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 07:04:47.351197 22382 ts_tablet_manager.cc:1397] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:04:47.351574 22381 tablet_bootstrap.cc:492] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: No bootstrap required, opened a new log
I20260620 07:04:47.351815 22381 ts_tablet_manager.cc:1397] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 07:04:47.358778 22380 raft_consensus.cc:348] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.359197 22380 raft_consensus.cc:374] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:47.359339 22380 raft_consensus.cc:729] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b72ef1ce0e5458fb38f1c6976c8b89f, State: Initialized, Role: FOLLOWER
I20260620 07:04:47.359963 22380 consensus_queue.cc:260] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.361555 22380 ts_tablet_manager.cc:1428] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260620 07:04:47.361905 22381 raft_consensus.cc:348] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.362417 22381 raft_consensus.cc:374] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:47.362522 22381 raft_consensus.cc:729] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb7cb0e516c4019ba7a3ce695882470, State: Initialized, Role: FOLLOWER
I20260620 07:04:47.362773 22382 raft_consensus.cc:348] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.363130 22381 consensus_queue.cc:260] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.363241 22382 raft_consensus.cc:374] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:04:47.363687 22382 raft_consensus.cc:729] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ecf55fee2e4743a156d16041682459, State: Initialized, Role: FOLLOWER
I20260620 07:04:47.364336 22382 consensus_queue.cc:260] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.367553 22382 ts_tablet_manager.cc:1428] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 07:04:47.368095 22229 heartbeater.cc:503] Master 127.11.81.254:40297 was elected leader, sending a full tablet report...
I20260620 07:04:47.368294 22362 heartbeater.cc:503] Master 127.11.81.254:40297 was elected leader, sending a full tablet report...
I20260620 07:04:47.369817 22381 ts_tablet_manager.cc:1428] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260620 07:04:47.371971 22388 raft_consensus.cc:482] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:04:47.372383 22388 raft_consensus.cc:504] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.374032 22388 leader_election.cc:288] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bb7cb0e516c4019ba7a3ce695882470 (127.11.81.194:34721), 3b72ef1ce0e5458fb38f1c6976c8b89f (127.11.81.193:42803)
I20260620 07:04:47.380270 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.374598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:51087 (local address 127.11.81.194:34721)
0620 07:04:47.374755 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.374759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.374774 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:04:47.374822 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.374825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.374963 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.375064 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:47.375461 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.376392 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.378335 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.379504 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.379613 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.379666 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.379993 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.380080 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:04:47.380451 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.374208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:34721 (local address 127.11.81.195:51087)
0620 07:04:47.374479 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.374495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.374598 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.375106 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.375110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.375142 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:47.375336 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.375347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.376522 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.376528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.378186 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.378201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.379639 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.379646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.379831 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.379865 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.379978 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.380112 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1}
I20260620 07:04:47.381407 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.374466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:35069 (local address 127.11.81.193:42803)
0620 07:04:47.374661 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.374667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.375421 (+ 754us) server_negotiation.cc:407] Connection header received
0620 07:04:47.375564 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.375568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.375623 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.375687 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:04:47.376183 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.377156 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.379316 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.380428 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.380597 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.380667 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.381050 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.381177 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:04:47.381487 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.374210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:42803 (local address 127.11.81.195:35069)
0620 07:04:47.375276 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0620 07:04:47.375333 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:04:47.375456 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:04:47.375716 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:04:47.375720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:04:47.375743 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:04:47.376065 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.376075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.377302 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.377307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.379129 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:04:47.379143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.380782 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.380787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:04:47.380900 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.380921 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:04:47.381004 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:04:47.381124 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":471,"thread_start_us":271,"threads_started":1}
I20260620 07:04:47.381397 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce50de71acb64c729db4b7f7765fa426" candidate_uuid: "59ecf55fee2e4743a156d16041682459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb7cb0e516c4019ba7a3ce695882470" is_pre_election: true
I20260620 07:04:47.381959 22184 raft_consensus.cc:2454] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ecf55fee2e4743a156d16041682459 in term 0.
I20260620 07:04:47.382401 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce50de71acb64c729db4b7f7765fa426" candidate_uuid: "59ecf55fee2e4743a156d16041682459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" is_pre_election: true
I20260620 07:04:47.382593 22251 leader_election.cc:302] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ecf55fee2e4743a156d16041682459, 7bb7cb0e516c4019ba7a3ce695882470; no voters:
I20260620 07:04:47.382890 22050 raft_consensus.cc:2454] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ecf55fee2e4743a156d16041682459 in term 0.
I20260620 07:04:47.382956 22388 raft_consensus.cc:2781] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:04:47.383062 22388 raft_consensus.cc:482] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:04:47.383155 22388 raft_consensus.cc:3037] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:47.385331 22388 raft_consensus.cc:504] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.385953 22388 leader_election.cc:288] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [CANDIDATE]: Term 1 election: Requested vote from peers 7bb7cb0e516c4019ba7a3ce695882470 (127.11.81.194:34721), 3b72ef1ce0e5458fb38f1c6976c8b89f (127.11.81.193:42803)
I20260620 07:04:47.386371 22184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce50de71acb64c729db4b7f7765fa426" candidate_uuid: "59ecf55fee2e4743a156d16041682459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb7cb0e516c4019ba7a3ce695882470"
I20260620 07:04:47.386420 22050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce50de71acb64c729db4b7f7765fa426" candidate_uuid: "59ecf55fee2e4743a156d16041682459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f"
I20260620 07:04:47.386584 22184 raft_consensus.cc:3037] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:47.386595 22050 raft_consensus.cc:3037] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:04:47.389679 22050 raft_consensus.cc:2454] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ecf55fee2e4743a156d16041682459 in term 1.
I20260620 07:04:47.389679 22184 raft_consensus.cc:2454] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ecf55fee2e4743a156d16041682459 in term 1.
I20260620 07:04:47.390098 22251 leader_election.cc:302] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b72ef1ce0e5458fb38f1c6976c8b89f, 59ecf55fee2e4743a156d16041682459, 7bb7cb0e516c4019ba7a3ce695882470; no voters:
I20260620 07:04:47.390373 22388 raft_consensus.cc:2781] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:04:47.390786 22388 raft_consensus.cc:686] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 1 LEADER]: Becoming Leader. State: Replica: 59ecf55fee2e4743a156d16041682459, State: Running, Role: LEADER
I20260620 07:04:47.391319 22388 consensus_queue.cc:237] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } }
I20260620 07:04:47.396006 21905 catalog_manager.cc:5495] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59ecf55fee2e4743a156d16041682459 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "59ecf55fee2e4743a156d16041682459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59ecf55fee2e4743a156d16041682459" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 37569 } health_report { overall_health: HEALTHY } } }
I20260620 07:04:47.425758 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.420680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.11.81.195:37569)
0620 07:04:47.420870 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.420874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.420891 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:04:47.421105 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.421109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.421194 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.421342 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:04:47.421740 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.422500 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.423814 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.424204 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.424294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.424718 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:04:47.425249 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:04:47.425356 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.425496 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.425562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:04:47.433840 22050 raft_consensus.cc:1261] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 1 FOLLOWER]: Refusing update from remote peer 59ecf55fee2e4743a156d16041682459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:47.433840 22184 raft_consensus.cc:1261] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 1 FOLLOWER]: Refusing update from remote peer 59ecf55fee2e4743a156d16041682459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:04:47.434782 22390 consensus_queue.cc:1035] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b72ef1ce0e5458fb38f1c6976c8b89f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:47.435091 22388 consensus_queue.cc:1035] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb7cb0e516c4019ba7a3ce695882470" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:04:47.446846 22394 mvcc.cc:205] Tried to move back new op lower bound from 7298822502119792640 to 7298822501958873088. Current Snapshot: MvccSnapshot[applied={T|T < 7298822502119792640}]
I20260620 07:04:47.458050 21906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:47.458293 21906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:04:47.461776 21906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0ca39635a04feda2d0d5b8501b7abc: Sending DeleteTablet for 3 replicas of tablet ce50de71acb64c729db4b7f7765fa426
I20260620 07:04:47.462910 22164 tablet_service.cc:1514] Processing DeleteTablet for tablet ce50de71acb64c729db4b7f7765fa426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:47 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:51812
I20260620 07:04:47.463150 22297 tablet_service.cc:1514] Processing DeleteTablet for tablet ce50de71acb64c729db4b7f7765fa426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:47 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:60718
I20260620 07:04:47.463575 22030 tablet_service.cc:1514] Processing DeleteTablet for tablet ce50de71acb64c729db4b7f7765fa426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:04:47 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:52952
I20260620 07:04:47.465401 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21967
I20260620 07:04:47.474543 22405 tablet_replica.cc:330] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: stopping tablet replica
I20260620 07:04:47.475131 22405 raft_consensus.cc:2229] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:47.475801 22405 raft_consensus.cc:2258] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:47.477860 22405 ts_tablet_manager.cc:1905] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:47.484954 22405 ts_tablet_manager.cc:1918] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:47.485224 22405 log.cc:1196] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/ce50de71acb64c729db4b7f7765fa426
I20260620 07:04:47.485836 22405 ts_tablet_manager.cc:1939] T ce50de71acb64c729db4b7f7765fa426 P 3b72ef1ce0e5458fb38f1c6976c8b89f: Deleting consensus metadata
I20260620 07:04:47.487934 21893 catalog_manager.cc:4841] TS 3b72ef1ce0e5458fb38f1c6976c8b89f (127.11.81.193:42803): tablet ce50de71acb64c729db4b7f7765fa426 (table test-table [id=9a67982cbd404a54a3aa468fff267376]) successfully deleted
I20260620 07:04:47.491956 22404 tablet_replica.cc:330] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: stopping tablet replica
I20260620 07:04:47.493250 22404 raft_consensus.cc:2229] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:04:47.494076 22404 raft_consensus.cc:2258] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:04:47.496398 22404 ts_tablet_manager.cc:1905] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:47.501015 22403 tablet_replica.cc:330] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: stopping tablet replica
I20260620 07:04:47.501495 22403 raft_consensus.cc:2229] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:04:47.502038 22403 raft_consensus.cc:2258] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 07:04:47 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:04:47.505204 22403 ts_tablet_manager.cc:1905] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:04:47.506852 22404 ts_tablet_manager.cc:1918] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:47.507035 22404 log.cc:1196] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/ce50de71acb64c729db4b7f7765fa426
I20260620 07:04:47.507612 22404 ts_tablet_manager.cc:1939] T ce50de71acb64c729db4b7f7765fa426 P 59ecf55fee2e4743a156d16041682459: Deleting consensus metadata
I20260620 07:04:47.509444 21892 catalog_manager.cc:4841] TS 59ecf55fee2e4743a156d16041682459 (127.11.81.195:37569): tablet ce50de71acb64c729db4b7f7765fa426 (table test-table [id=9a67982cbd404a54a3aa468fff267376]) successfully deleted
I20260620 07:04:47.511132 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.467652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.11.81.193:42803)
0620 07:04:47.467823 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.467829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.492644 (+ 24815us) server_negotiation.cc:407] Connection header received
0620 07:04:47.492861 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.492867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.492992 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.493094 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:47.495389 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.496593 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.498598 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.499300 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.499402 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.503911 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:47.503930 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:47.503960 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:47.504003 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:47.507533 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.508403 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.508413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.508416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.508568 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.509091 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.509096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.509099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.509570 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:47.509867 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.510345 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.510587 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:04:47.511737 22090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:47.512977 22403 ts_tablet_manager.cc:1918] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:04:47.513176 22403 log.cc:1196] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/ce50de71acb64c729db4b7f7765fa426
I20260620 07:04:47.513762 22403 ts_tablet_manager.cc:1939] T ce50de71acb64c729db4b7f7765fa426 P 7bb7cb0e516c4019ba7a3ce695882470: Deleting consensus metadata
I20260620 07:04:47.515771 21891 catalog_manager.cc:4841] TS 7bb7cb0e516c4019ba7a3ce695882470 (127.11.81.194:34721): tablet ce50de71acb64c729db4b7f7765fa426 (table test-table [id=9a67982cbd404a54a3aa468fff267376]) successfully deleted
I20260620 07:04:47.600157 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21967
I20260620 07:04:47.619112 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 22101
Jun 20 07:04:47 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:04:47.637151 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.619969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.11.81.194:34721)
0620 07:04:47.620173 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.620180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.620202 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:04:47.620331 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.620337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.620429 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.620534 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:47.622074 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.623182 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.625216 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.625860 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.625990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.630726 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:47.630747 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:47.630767 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:47.630815 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:47.634323 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.635041 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.635051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.635055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.635200 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.635571 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.635576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.635579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.636067 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:47.636353 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.636569 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.636792 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:04:47.637861 22224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:47.724625 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 22101
I20260620 07:04:47.738478 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 22234
Jun 20 07:04:47 dist-test-slave-wvz3 krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:04:47.756520 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:04:47.739227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.11.81.195:37569)
0620 07:04:47.739531 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 07:04:47.739535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:04:47.739660 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:04:47.739822 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:04:47.739826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:04:47.739916 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:04:47.740021 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:04:47.741764 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.742610 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.744439 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:04:47.744864 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:04:47.744962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:04:47.750325 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:04:47.750340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:04:47.750381 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:04:47.750440 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:04:47.753292 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.754258 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.754265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.754267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.754473 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:04:47.754888 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:04:47.754892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:04:47.754894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:04:47.755264 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:04:47.755553 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 07:04:47.755910 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 07:04:47.756086 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260620 07:04:47.757153 22357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:47.879995 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 22234
I20260620 07:04:47.901767 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 21874
I20260620 07:04:47.902935 21936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:04:48.016340 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 21874
2026-06-20T07:04:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4073 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:04:48 dist-test-slave-wvz3 krb5kdc[22413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:04:48 dist-test-slave-wvz3 krb5kdc[22413](info): set up 2 sockets
Jun 20 07:04:48 dist-test-slave-wvz3 krb5kdc[22413](info): commencing operation
krb5kdc: starting...
W20260620 07:04:49.040098 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:04:49 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:04:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangeradmin_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangeradmin_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangerlookup_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangerlookup_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
I20260620 07:04:49.358716 11591 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOWNMWE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/postgres -l logfile start
2026-06-20 07:04:51.810 UTC [22443] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:04:51.810 UTC [22443] LOG: listening on IPv4 address "127.11.81.212", port 58619
2026-06-20 07:04:51.813 UTC [22443] LOG: listening on Unix socket "/tmp/.s.PGSQL.58619"
2026-06-20 07:04:51.832 UTC [22446] LOG: database system was shut down at 2026-06-20 07:04:50 UTC
2026-06-20 07:04:51.836 UTC [22443] LOG: database system is ready to accept connections
I20260620 07:04:52.731593 11591 mini_postgres.cc:96] Postgres bound to 58619
2026-06-20 07:04:52.748 UTC [22454] FATAL: database "slave" does not exist
127.11.81.212:58619 - accepting connections
I20260620 07:04:52.749092 11591 mini_ranger.cc:161] Starting Ranger...
I20260620 07:04:52.774860 11591 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:04:53.237875 11591 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:04:53.237999 11591 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin
2026-06-20 07:04:53,533 [I] DB FLAVOR :POSTGRES
2026-06-20 07:04:53,534 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:04:53,534 [I] Checking connection..
2026-06-20 07:04:53,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:04:53,792 [I] Checking connection passed.
2026-06-20 07:04:53,792 [I] --------- Verifying version history table ---------
2026-06-20 07:04:53,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:04:54,041 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:04:54,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:04:54,293 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:04:54,294 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:04:54,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:04:54.520 UTC [22518] WARNING: there is no transaction in progress
2026-06-20 07:04:54,526 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:04:54,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:04:54,756 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:04:54,757 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:04:54,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:04:55,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:04:55,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvz3.c.gcp-upstream.internal','N') ;"
2026-06-20 07:04:55,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:04:55,739 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:04:55,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:04:55,976 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:04:55,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:04:56,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:04:56,467 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:04:56,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:04:57.452 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.469 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.486 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.501 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.517 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.536 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.601 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.609 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.623 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.634 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.644 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.659 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.669 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.676 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.682 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:57.689 UTC [22626] WARNING: there is no transaction in progress
2026-06-20 07:04:58,313 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:04:58,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:04:58,563 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:04:58,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:04:58,821 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:04:58,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:04:59,068 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:04:59,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:04:59,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:04:59,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvz3.c.gcp-upstream.internal';"
2026-06-20 07:04:59,784 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:04:59,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:05:00,042 [I] DB_PATCHES have already been applied
I20260620 07:05:00.048264 11591 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOWNMWE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:05:00.048365 11591 mini_ranger.cc:193] Using host: 127.11.81.212
I20260620 07:05:00.063366 11591 mini_ranger.cc:227] Ranger admin URL: http://127.11.81.212:34589
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:05:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:05:01.085125 11591 mini_ranger.cc:160] Time spent starting Ranger: real 8.336s user 0.000s sys 0.043s
Jun 20, 2026 7:05:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:05:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:05:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:05:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:05:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:05:22.073587 11591 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangerkms_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/rangerkms_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.81.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.81.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:05:22 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939122, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:05:22.231223 11591 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:05:22.251924 11591 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:05:23.107729 11591 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:05:23.107914 11591 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms
I20260620 07:05:23.108049 11591 mini_ranger_kms.cc:227] Using postgres at 127.11.81.212:58619
2026-06-20 07:05:23,510 [I] DB FLAVOR :POSTGRES
2026-06-20 07:05:23,510 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:05:23,510 [I] Checking connection
2026-06-20 07:05:23,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:05:23,746 [I] connection success
2026-06-20 07:05:23,746 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:05:23,746 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:05:23,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:05:23,980 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:05:23,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:05:23,980 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:05:23,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:58619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:05:24,246 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:05:24.711627 11591 mini_ranger_kms.cc:310] Created kms service
I20260620 07:05:24.941617 11591 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:05:24.987093 11591 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:05:24 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939122, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:25.387320 11591 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:05:25.387683 11591 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:05:25.387758 11591 mini_ranger_kms.cc:240] Using host: 127.11.81.212
I20260620 07:05:25.408903 11591 mini_ranger_kms.cc:276] Ranger KMS PID: 22806
I20260620 07:05:25.409082 11591 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.81.212:44143
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:05:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:05:27.489276 11591 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.258s user 0.103s sys 0.326s
I20260620 07:05:27.489485 11591 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:05:27.489684 11591 mini_ranger_kms.cc:401] 127.11.81.212:44143/kms/v1/keys
Jun 20 07:05:27 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:05:28 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
Jun 20 07:05:28 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
Jun 20 07:05:30 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:05:30.980247 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39889
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39889
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:44143/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:05:31.324725 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:05:31.325031 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:05:31.325160 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:05:31.332830 22857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:05:31.332962 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:05:31.333040 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:05:31.333114 22857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:05:31.333175 22857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:05:31.343622 22857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.81.212:44143/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39889
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:05:31.345011 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:05:31.346402 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:05:31.360508 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:31.360522 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:31.360682 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:05:31.361991 22857 server_base.cc:1034] running on GCE node
I20260620 07:05:31.362821 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:05:31.364481 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:05:31.365736 22857 hybrid_clock.cc:648] HybridClock initialized: now 1781939131365683 us; error 48 us; skew 500 ppm
I20260620 07:05:31.366079 22857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:05:31 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:05:31.369547 22857 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:05:31.371553 22857 webserver.cc:460] Webserver started at http://127.11.81.254:41065/ using document root <none> and password file <none>
I20260620 07:05:31.372448 22857 fs_manager.cc:359] Metadata directory not provided
I20260620 07:05:31.372648 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:05:31.373121 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:05:31 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939131, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:31.424528 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "ec0833143569497192d6b8618a0d47e5"
format_stamp: "Formatted at 2026-06-20 07:05:31 on dist-test-slave-wvz3"
server_key: "7ae941a28e6de772509e9fa1a582f148"
server_key_iv: "4057cb9c148811d54307bbe88c3b00c0"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:31.425704 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "ec0833143569497192d6b8618a0d47e5"
format_stamp: "Formatted at 2026-06-20 07:05:31 on dist-test-slave-wvz3"
server_key: "7ae941a28e6de772509e9fa1a582f148"
server_key_iv: "4057cb9c148811d54307bbe88c3b00c0"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:31.433197 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 07:05:31.454433 22872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:31.456440 22857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:05:31.456661 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "ec0833143569497192d6b8618a0d47e5"
format_stamp: "Formatted at 2026-06-20 07:05:31 on dist-test-slave-wvz3"
server_key: "7ae941a28e6de772509e9fa1a582f148"
server_key_iv: "4057cb9c148811d54307bbe88c3b00c0"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:31.456941 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:05:31.500437 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:05:31.505709 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:05:31.506035 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:05:31.525182 22857 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:39889
I20260620 07:05:31.525198 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:39889 every 8 connection(s)
I20260620 07:05:31.526903 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:05:31.530875 22924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:05:31.531343 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 22857
I20260620 07:05:31.531551 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:05:31.532337 11591 external_mini_cluster.cc:1428] Setting key 50c36b88a447cd587ab4b58b8fa8db62
I20260620 07:05:31.546804 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: Bootstrap starting.
I20260620 07:05:31.550273 22924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: Neither blocks nor log segments found. Creating new log.
I20260620 07:05:31.551767 22924 log.cc:824] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:05:31.555449 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: No bootstrap required, opened a new log
Jun 20 07:05:31 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:05:31.565487 22924 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } }
I20260620 07:05:31.565979 22924 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:05:31.566083 22924 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0833143569497192d6b8618a0d47e5, State: Initialized, Role: FOLLOWER
I20260620 07:05:31.566797 22924 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } }
I20260620 07:05:31.567076 22924 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:05:31.567198 22924 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:05:31.567409 22924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:05:31.570621 22924 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } }
I20260620 07:05:31.571208 22924 leader_election.cc:302] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0833143569497192d6b8618a0d47e5; no voters:
I20260620 07:05:31.571895 22924 leader_election.cc:288] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:05:31.572189 22929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:05:31.573550 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:31.534620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.11.81.254:39889)
0620 07:05:31.546339 (+ 11719us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:31.546368 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:31.546427 (+ 59us) server_negotiation.cc:407] Connection header received
0620 07:05:31.548809 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:31.548841 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:31.549453 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:31.550104 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:31.551696 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:31.554552 (+ 2856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:31.556427 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:31.557470 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:31.557661 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:31.563654 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:31.563709 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:31.563741 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:31.563795 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:31.567631 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:31.568550 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:31.568559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:31.568568 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:31.568756 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:31.569216 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:31.569222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:31.569225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:31.569950 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:31.571410 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:31.571695 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 07:05:31.572134 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11440,"thread_start_us":12265,"threads_started":1}
I20260620 07:05:31.577817 22929 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [term 1 LEADER]: Becoming Leader. State: Replica: ec0833143569497192d6b8618a0d47e5, State: Running, Role: LEADER
I20260620 07:05:31.578474 22929 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } }
I20260620 07:05:31.582541 22924 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:05:31.584599 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0833143569497192d6b8618a0d47e5. Latest consensus state: current_term: 1 leader_uuid: "ec0833143569497192d6b8618a0d47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } } }
I20260620 07:05:31.584635 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0833143569497192d6b8618a0d47e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0833143569497192d6b8618a0d47e5" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39889 } } }
I20260620 07:05:31.585162 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:05:31.585192 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:05:31.591400 22934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:05:31.597999 22934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:05:31.617861 22934 catalog_manager.cc:1347] Generated new cluster ID: 5d98848b6b104e80b5f9562a692582fd
I20260620 07:05:31.617996 22934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:05:31.635223 22934 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:05:31.637310 22934 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:05:31.654218 22924 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:05:31.677762 22934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: Generated new TSK 0
I20260620 07:05:31.679849 22934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:05:32.559769 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:44143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:05:32 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939132, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:05:32.982728 22969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:05:32.983273 22969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:05:32.983501 22969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:05:32.995483 22969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:05:32.995718 22969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:05:32.995981 22969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:05:33.012531 22969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.81.212:44143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:05:33.014364 22969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:05:33.016186 22969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:05:33.034448 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:33.035545 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:33.040452 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:05:33.040802 22969 server_base.cc:1034] running on GCE node
I20260620 07:05:33.041837 22969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:05:33.049818 22969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:05:33.053349 22969 hybrid_clock.cc:648] HybridClock initialized: now 1781939133053233 us; error 107 us; skew 500 ppm
I20260620 07:05:33.053911 22969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:05:33 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939133, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:05:33.058341 22969 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:05:33.070971 22969 webserver.cc:460] Webserver started at http://127.11.81.193:38077/ using document root <none> and password file <none>
I20260620 07:05:33.072222 22969 fs_manager.cc:359] Metadata directory not provided
I20260620 07:05:33.072446 22969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:05:33.072937 22969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:05:33 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939133, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:33.092250 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "01f34263bfdb41139579e379707442c7"
format_stamp: "Formatted at 2026-06-20 07:05:33 on dist-test-slave-wvz3"
server_key: "4784e1bb788366e48df77d00c9c0c4b5"
server_key_iv: "038fe2bdba165bea2ca1b350e7d865bf"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:33.093472 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "01f34263bfdb41139579e379707442c7"
format_stamp: "Formatted at 2026-06-20 07:05:33 on dist-test-slave-wvz3"
server_key: "4784e1bb788366e48df77d00c9c0c4b5"
server_key_iv: "038fe2bdba165bea2ca1b350e7d865bf"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:33.100344 22969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 07:05:33.140064 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:33.142208 22969 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.006s sys 0.008s
I20260620 07:05:33.142508 22969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "01f34263bfdb41139579e379707442c7"
format_stamp: "Formatted at 2026-06-20 07:05:33 on dist-test-slave-wvz3"
server_key: "4784e1bb788366e48df77d00c9c0c4b5"
server_key_iv: "038fe2bdba165bea2ca1b350e7d865bf"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:33.142869 22969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:05:33.193251 22969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:05:33.199033 22969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:05:33.199504 22969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:05:33.207187 22969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:05:33.215672 22969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:05:33.215878 22969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:33.216075 22969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:05:33.216184 22969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:05:33 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939132, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:33.500303 22969 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:38923
I20260620 07:05:33.500322 23101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:38923 every 8 connection(s)
I20260620 07:05:33.501940 22969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:05:33.504503 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 22969
I20260620 07:05:33.504704 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:05:33.505470 11591 external_mini_cluster.cc:1428] Setting key 6daecb9152a94ccea7dd572ae3eaee9f
Jun 20 07:05:33 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939133, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:05:33.533047 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:33.504764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:59197 (local address 127.11.81.254:39889)
0620 07:05:33.505245 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:33.505251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:33.506501 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 07:05:33.509202 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:33.509207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:33.509304 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:33.509417 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:33.511603 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:33.512701 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:33.515510 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:33.516143 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:33.516275 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:33.520718 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:33.520725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:33.520729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:33.520782 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:33.524019 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:33.530081 (+ 6062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:33.530088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:33.530092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:33.530264 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:33.530843 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:33.530848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:33.530851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:33.531204 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:33.531367 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:33.532399 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0620 07:05:33.532656 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260620 07:05:33.535055 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:33.505493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39889 (local address 127.11.81.193:59197)
0620 07:05:33.506324 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:33.506375 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:33.508844 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:33.509723 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:33.509735 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:33.510365 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:05:33.511380 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:33.511413 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:33.512947 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:33.512954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:33.515316 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:33.515337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:33.516421 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:33.516429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:33.516742 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:33.517563 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:05:33.517600 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:05:33.520418 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:05:33.529214 (+ 8796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:33.529226 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:33.529249 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:33.529914 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:33.530437 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:33.530442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:33.530446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:33.530703 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:33.531456 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:05:33.531464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:05:33.531985 (+ 521us) client_negotiation.cc:769] Sending connection context
0620 07:05:33.533500 (+ 1515us) client_negotiation.cc:240] Negotiation successful
0620 07:05:33.533952 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":321,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:05:33.537431 23102 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39889
I20260620 07:05:33.537739 23102 heartbeater.cc:464] Registering TS with master...
I20260620 07:05:33.538573 23102 heartbeater.cc:511] Master 127.11.81.254:39889 requested a full tablet report, sending...
I20260620 07:05:33.541371 22889 ts_manager.cc:194] Registered new tserver with Master: 01f34263bfdb41139579e379707442c7 (127.11.81.193:38923)
I20260620 07:05:33.543249 22889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:59197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:05:33.566758 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:44143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:05:33 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939132, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
W20260620 07:05:33.988214 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:05:33.988519 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:05:33.988679 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:05:33.996064 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:05:33.996203 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:05:33.996361 23108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:05:34.006582 23108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.81.212:44143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:05:34.008028 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:05:34.009397 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:05:34.025502 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:34.032239 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:34.033983 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:05:34.034135 23108 server_base.cc:1034] running on GCE node
I20260620 07:05:34.034970 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:05:34.036293 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:05:34.037573 23108 hybrid_clock.cc:648] HybridClock initialized: now 1781939134037507 us; error 53 us; skew 500 ppm
I20260620 07:05:34.037962 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:05:34.042107 23108 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:05:34.044026 23108 webserver.cc:460] Webserver started at http://127.11.81.194:45465/ using document root <none> and password file <none>
I20260620 07:05:34.044906 23108 fs_manager.cc:359] Metadata directory not provided
I20260620 07:05:34.045032 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:05:34.045367 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:34.067539 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "fc51950a38fb423c8ccc1c24667c7150"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "b2d344e988e486c7b5d528e1f3c43df0"
server_key_iv: "fc86ce45416162e983c444e3bc292fb4"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.068737 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "fc51950a38fb423c8ccc1c24667c7150"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "b2d344e988e486c7b5d528e1f3c43df0"
server_key_iv: "fc86ce45416162e983c444e3bc292fb4"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.089474 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.016s sys 0.002s
I20260620 07:05:34.123525 23132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.125698 23108 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.009s
I20260620 07:05:34.125916 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "fc51950a38fb423c8ccc1c24667c7150"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "b2d344e988e486c7b5d528e1f3c43df0"
server_key_iv: "fc86ce45416162e983c444e3bc292fb4"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.126190 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:05:34.144575 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:05:34.149915 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:05:34.150249 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:05:34.151453 23108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:05:34.153345 23108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:05:34.153487 23108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.153620 23108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:05:34.153674 23108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.196409 23108 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36087
I20260620 07:05:34.196445 23244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36087 every 8 connection(s)
I20260620 07:05:34.198190 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:05:34.203965 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 23108
I20260620 07:05:34.204188 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:05:34.204782 11591 external_mini_cluster.cc:1428] Setting key 98f96ec3a2ceaced9fff02cbd9ee17da
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
I20260620 07:05:34.246191 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.201280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:51937 (local address 127.11.81.254:39889)
0620 07:05:34.227400 (+ 26120us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.227406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.227427 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:05:34.227487 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.227491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.227568 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.227666 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:34.230322 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.231451 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.233872 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.234491 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.234608 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.239024 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:34.239032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:34.239036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:34.239075 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:34.242383 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.243382 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.243388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.243391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.243545 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.244063 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.244068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.244071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.244470 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:34.244585 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.245574 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.245805 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25929,"thread_start_us":25807,"threads_started":1}
I20260620 07:05:34.248605 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.202081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39889 (local address 127.11.81.194:51937)
0620 07:05:34.224051 (+ 21970us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.224115 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.226539 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.228133 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.228147 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.228826 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:05:34.230101 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.230129 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.231663 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.231668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.233671 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.233699 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.234971 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.234979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.235323 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.236135 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:05:34.236165 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:05:34.238773 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:05:34.242589 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:34.242599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:34.242622 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:34.243202 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:34.243696 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:34.243700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:34.243704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:34.243943 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:34.244688 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:05:34.244695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:05:34.245151 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.246528 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.247078 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21590,"thread_start_us":21353,"threads_started":1}
I20260620 07:05:34.250496 23245 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39889
I20260620 07:05:34.250825 23245 heartbeater.cc:464] Registering TS with master...
I20260620 07:05:34.251680 23245 heartbeater.cc:511] Master 127.11.81.254:39889 requested a full tablet report, sending...
I20260620 07:05:34.253389 22889 ts_manager.cc:194] Registered new tserver with Master: fc51950a38fb423c8ccc1c24667c7150 (127.11.81.194:36087)
I20260620 07:05:34.254706 22889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:51937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:05:34.272212 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:44143/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:05:34.546996 23102 heartbeater.cc:503] Master 127.11.81.254:39889 was elected leader, sending a full tablet report...
W20260620 07:05:34.600625 23251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:05:34.600930 23251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:05:34.601019 23251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:05:34.608368 23251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:05:34.608489 23251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:05:34.608598 23251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:05:34.618371 23251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.81.212:44143/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:05:34.619628 23251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:05:34.620841 23251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:05:34.634282 23256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:34.634229 23257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:05:34.634229 23259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:05:34.634781 23251 server_base.cc:1034] running on GCE node
I20260620 07:05:34.635468 23251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:05:34.636382 23251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:05:34.637581 23251 hybrid_clock.cc:648] HybridClock initialized: now 1781939134637539 us; error 37 us; skew 500 ppm
I20260620 07:05:34.637871 23251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:05:34.641026 23251 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:05:34.642566 23251 webserver.cc:460] Webserver started at http://127.11.81.195:44749/ using document root <none> and password file <none>
I20260620 07:05:34.643235 23251 fs_manager.cc:359] Metadata directory not provided
I20260620 07:05:34.643373 23251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:05:34.643663 23251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:05:34.660495 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "7797f02a0d6a432ea30fdc206829c45d"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "f01e33001b93211156506fb1c6168680"
server_key_iv: "b82e63e4e913f47b79320d5fcca8dbdd"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.661389 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "7797f02a0d6a432ea30fdc206829c45d"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "f01e33001b93211156506fb1c6168680"
server_key_iv: "b82e63e4e913f47b79320d5fcca8dbdd"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.667791 23251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 07:05:34.684815 23266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.686415 23251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:05:34.686581 23251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "7797f02a0d6a432ea30fdc206829c45d"
format_stamp: "Formatted at 2026-06-20 07:05:34 on dist-test-slave-wvz3"
server_key: "f01e33001b93211156506fb1c6168680"
server_key_iv: "b82e63e4e913f47b79320d5fcca8dbdd"
server_key_version: "kuduclusterkey@0"
I20260620 07:05:34.686810 23251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:05:34.725613 23251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:05:34.729614 23251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:05:34.729918 23251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:05:34.730890 23251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:05:34.732406 23251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:05:34.732502 23251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.732618 23251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:05:34.732682 23251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:05:34.763080 23251 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33299
I20260620 07:05:34.763120 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33299 every 8 connection(s)
I20260620 07:05:34.764397 23251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:05:34.774691 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 23251
I20260620 07:05:34.774876 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:05:34.775312 11591 external_mini_cluster.cc:1428] Setting key da34192a31b90b3b7c7a459bec3cacaa
Jun 20 07:05:34 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939134, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:05:34.783644 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.766973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:56953 (local address 127.11.81.254:39889)
0620 07:05:34.767461 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.767466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.768372 (+ 906us) server_negotiation.cc:407] Connection header received
0620 07:05:34.770137 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.770151 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.770231 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.770325 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:34.771835 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.772601 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.774116 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.774556 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.774669 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.778488 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:34.778495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:34.778499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:34.778535 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:34.780918 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.781567 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.781572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.781574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.781694 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.782083 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.782086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.782088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.782347 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:34.782450 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.783177 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.783384 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":235,"threads_started":1}
I20260620 07:05:34.784597 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.767526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39889 (local address 127.11.81.195:56953)
0620 07:05:34.768221 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.768258 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.769864 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.770578 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.770595 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.771038 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:05:34.771670 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.771688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.772748 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.772753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.773970 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.773990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.774777 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.774783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.775047 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.775760 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:05:34.775788 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:05:34.778222 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:05:34.781034 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:34.781041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:34.781054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:34.781469 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:34.781801 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:05:34.781805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:05:34.781807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:05:34.781999 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:05:34.782450 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:05:34.782457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:05:34.782828 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.783059 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.783540 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":270,"threads_started":1}
I20260620 07:05:34.785851 23379 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39889
I20260620 07:05:34.786127 23379 heartbeater.cc:464] Registering TS with master...
I20260620 07:05:34.786872 23379 heartbeater.cc:511] Master 127.11.81.254:39889 requested a full tablet report, sending...
I20260620 07:05:34.788195 22889 ts_manager.cc:194] Registered new tserver with Master: 7797f02a0d6a432ea30fdc206829c45d (127.11.81.195:33299)
I20260620 07:05:34.789113 22889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:56953
I20260620 07:05:34.791157 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:05:34.809041 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.795805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.11.81.254:39889)
0620 07:05:34.795994 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.796000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.796541 (+ 541us) server_negotiation.cc:407] Connection header received
0620 07:05:34.796762 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.796766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.796846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.796933 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:34.798413 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.799202 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.800618 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.801078 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.801176 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.803835 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:34.803840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:34.803844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:34.803879 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:34.806457 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.807170 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.807174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.807176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.807348 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:34.807751 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:34.807756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:34.807759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:34.808046 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:34.808244 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.808578 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.808807 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:05:34.813426 22889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:05:34.820500 22889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:05:34.845963 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.833651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:38923 (local address 127.0.0.1:37178)
0620 07:05:34.837676 (+ 4025us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.837693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.837815 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.838492 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.838500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.838531 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:05:34.839023 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.839032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840654 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.840659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.843181 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.843198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.844975 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.844982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.845180 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.845238 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.845394 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.845598 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"mutex_wait_us":318,"spinlock_wait_cycles":725504,"thread_start_us":3369,"threads_started":1}
I20260620 07:05:34.846436 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.835916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.11.81.195:33299)
0620 07:05:34.836614 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.836622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.837742 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 07:05:34.837997 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.838005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.838239 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.838406 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:05:34.839202 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840659 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.843516 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.844929 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.845053 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.845272 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.845972 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.846137 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":356,"threads_started":1}
I20260620 07:05:34.846450 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.836622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33299 (local address 127.0.0.1:56148)
0620 07:05:34.837623 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.837641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.837831 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.838412 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.838417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.838443 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:05:34.839014 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.839026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840827 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.840835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.843362 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.843375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.845441 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.845448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.845703 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.845797 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.845894 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.846053 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":546,"spinlock_wait_cycles":14976,"thread_start_us":300,"threads_started":1}
I20260620 07:05:34.846479 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.833741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.11.81.194:36087)
0620 07:05:34.835979 (+ 2238us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.835988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.837193 (+ 1205us) server_negotiation.cc:407] Connection header received
0620 07:05:34.837570 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.837579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.837857 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.838047 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:05:34.838695 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840261 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.843420 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.844956 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.845066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.845248 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.846014 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.846186 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":1972,"threads_started":1}
I20260620 07:05:34.847319 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.834083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.11.81.193:38923)
0620 07:05:34.834775 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.834784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.837938 (+ 3154us) server_negotiation.cc:407] Connection header received
0620 07:05:34.838038 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.838046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.838282 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.838750 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:05:34.839215 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840499 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.843372 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.844459 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.844590 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.844837 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.846873 (+ 2036us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.847030 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":360,"threads_started":1}
I20260620 07:05:34.847797 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.833646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36087 (local address 127.0.0.1:34960)
0620 07:05:34.837022 (+ 3376us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.837075 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.837272 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.838047 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.838055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.838081 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:05:34.838475 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.838494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.840442 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.840448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.843233 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.843243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.845422 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.845428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.845725 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.845799 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.845894 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.847453 (+ 1559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3085,"mutex_wait_us":804,"thread_start_us":1649,"threads_started":1}
I20260620 07:05:34.850502 23037 tablet_service.cc:1467] Processing CreateTablet for tablet 19b8b911fd634a34a42f988bd1732486 (DEFAULT_TABLE table=test-table [id=44125684ef784437ba978db9f3e6a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:05:34.850575 23314 tablet_service.cc:1467] Processing CreateTablet for tablet 19b8b911fd634a34a42f988bd1732486 (DEFAULT_TABLE table=test-table [id=44125684ef784437ba978db9f3e6a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:05:34.851102 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 19b8b911fd634a34a42f988bd1732486 (DEFAULT_TABLE table=test-table [id=44125684ef784437ba978db9f3e6a58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:05:34.851934 23037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b8b911fd634a34a42f988bd1732486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:05:34.852043 23314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b8b911fd634a34a42f988bd1732486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:05:34.852309 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b8b911fd634a34a42f988bd1732486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:05:34.863593 23398 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Bootstrap starting.
I20260620 07:05:34.868024 23399 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Bootstrap starting.
I20260620 07:05:34.868479 23398 tablet_bootstrap.cc:654] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Neither blocks nor log segments found. Creating new log.
I20260620 07:05:34.869957 23398 log.cc:824] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:05:34.871500 23400 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Bootstrap starting.
I20260620 07:05:34.871738 23399 tablet_bootstrap.cc:654] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Neither blocks nor log segments found. Creating new log.
I20260620 07:05:34.873078 23399 log.cc:824] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Log is configured to *not* fsync() on all Append() calls
I20260620 07:05:34.873870 23398 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: No bootstrap required, opened a new log
I20260620 07:05:34.874168 23398 ts_tablet_manager.cc:1397] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 07:05:34.874878 23400 tablet_bootstrap.cc:654] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Neither blocks nor log segments found. Creating new log.
I20260620 07:05:34.875952 23400 log.cc:824] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:05:34.877390 23399 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: No bootstrap required, opened a new log
I20260620 07:05:34.877671 23399 ts_tablet_manager.cc:1397] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 07:05:34.878986 23400 tablet_bootstrap.cc:492] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: No bootstrap required, opened a new log
I20260620 07:05:34.879233 23400 ts_tablet_manager.cc:1397] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:05:34.881944 23398 raft_consensus.cc:348] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.882313 23398 raft_consensus.cc:374] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:05:34.882402 23398 raft_consensus.cc:729] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f34263bfdb41139579e379707442c7, State: Initialized, Role: FOLLOWER
I20260620 07:05:34.883008 23398 consensus_queue.cc:260] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.884770 23398 ts_tablet_manager.cc:1428] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:05:34.888716 23399 raft_consensus.cc:348] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.889212 23399 raft_consensus.cc:374] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:05:34.889315 23399 raft_consensus.cc:729] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc51950a38fb423c8ccc1c24667c7150, State: Initialized, Role: FOLLOWER
I20260620 07:05:34.889302 23400 raft_consensus.cc:348] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.889693 23400 raft_consensus.cc:374] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:05:34.889796 23400 raft_consensus.cc:729] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7797f02a0d6a432ea30fdc206829c45d, State: Initialized, Role: FOLLOWER
I20260620 07:05:34.890082 23399 consensus_queue.cc:260] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.890477 23400 consensus_queue.cc:260] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.891196 23245 heartbeater.cc:503] Master 127.11.81.254:39889 was elected leader, sending a full tablet report...
I20260620 07:05:34.891633 23379 heartbeater.cc:503] Master 127.11.81.254:39889 was elected leader, sending a full tablet report...
I20260620 07:05:34.892040 23399 ts_tablet_manager.cc:1428] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260620 07:05:34.892174 23400 ts_tablet_manager.cc:1428] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 07:05:34.925453 23404 raft_consensus.cc:482] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:05:34.925767 23404 raft_consensus.cc:504] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.927547 23404 leader_election.cc:288] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc51950a38fb423c8ccc1c24667c7150 (127.11.81.194:36087), 7797f02a0d6a432ea30fdc206829c45d (127.11.81.195:33299)
I20260620 07:05:34.934087 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.928100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:51033 (local address 127.11.81.195:33299)
0620 07:05:34.928314 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.928320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.928865 (+ 545us) server_negotiation.cc:407] Connection header received
0620 07:05:34.929046 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.929050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.929106 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.929170 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:05:34.929762 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.930509 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.932403 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.933242 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.933396 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.933469 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.933728 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.933851 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:05:34.934203 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.927900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33299 (local address 127.11.81.193:51033)
0620 07:05:34.928732 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.928752 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.928905 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.929216 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.929221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.929244 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:05:34.929615 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.929628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.930634 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.930639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.932270 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.932282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.933446 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.933452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.933594 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.933618 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.933695 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.933828 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":5,"thread_start_us":470,"threads_started":1}
I20260620 07:05:34.934418 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.927993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:35203 (local address 127.11.81.194:36087)
0620 07:05:34.928194 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.928201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.928904 (+ 703us) server_negotiation.cc:407] Connection header received
0620 07:05:34.929054 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.929058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.929124 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.929228 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:05:34.929803 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.930552 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.932467 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.933284 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.933435 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.933507 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.934052 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.934183 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:05:34.935138 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.928043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36087 (local address 127.11.81.193:35203)
0620 07:05:34.928779 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 07:05:34.928799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:05:34.928935 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:05:34.929253 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:05:34.929258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:05:34.929287 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:05:34.929659 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.929668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.930700 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.930703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.932317 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:05:34.932330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.933503 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.933509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:05:34.933635 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.933664 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:05:34.934637 (+ 973us) client_negotiation.cc:240] Negotiation successful
0620 07:05:34.934794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":37,"thread_start_us":462,"threads_started":1}
I20260620 07:05:34.935245 23334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b8b911fd634a34a42f988bd1732486" candidate_uuid: "01f34263bfdb41139579e379707442c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7797f02a0d6a432ea30fdc206829c45d" is_pre_election: true
I20260620 07:05:34.935834 23334 raft_consensus.cc:2454] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f34263bfdb41139579e379707442c7 in term 0.
I20260620 07:05:34.936239 23200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b8b911fd634a34a42f988bd1732486" candidate_uuid: "01f34263bfdb41139579e379707442c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc51950a38fb423c8ccc1c24667c7150" is_pre_election: true
I20260620 07:05:34.936570 22992 leader_election.cc:302] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f34263bfdb41139579e379707442c7, 7797f02a0d6a432ea30fdc206829c45d; no voters:
I20260620 07:05:34.936741 23200 raft_consensus.cc:2454] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f34263bfdb41139579e379707442c7 in term 0.
I20260620 07:05:34.937036 23404 raft_consensus.cc:2781] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:05:34.937193 23404 raft_consensus.cc:482] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:05:34.937319 23404 raft_consensus.cc:3037] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:05:34.939566 23404 raft_consensus.cc:504] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.940151 23404 leader_election.cc:288] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [CANDIDATE]: Term 1 election: Requested vote from peers fc51950a38fb423c8ccc1c24667c7150 (127.11.81.194:36087), 7797f02a0d6a432ea30fdc206829c45d (127.11.81.195:33299)
I20260620 07:05:34.940582 23200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b8b911fd634a34a42f988bd1732486" candidate_uuid: "01f34263bfdb41139579e379707442c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc51950a38fb423c8ccc1c24667c7150"
I20260620 07:05:34.940644 23334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b8b911fd634a34a42f988bd1732486" candidate_uuid: "01f34263bfdb41139579e379707442c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7797f02a0d6a432ea30fdc206829c45d"
I20260620 07:05:34.940747 23200 raft_consensus.cc:3037] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:05:34.940853 23334 raft_consensus.cc:3037] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:05:34.943022 23200 raft_consensus.cc:2454] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f34263bfdb41139579e379707442c7 in term 1.
I20260620 07:05:34.943056 23334 raft_consensus.cc:2454] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f34263bfdb41139579e379707442c7 in term 1.
I20260620 07:05:34.943435 22990 leader_election.cc:302] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f34263bfdb41139579e379707442c7, fc51950a38fb423c8ccc1c24667c7150; no voters:
I20260620 07:05:34.943740 23404 raft_consensus.cc:2781] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:05:34.944169 23404 raft_consensus.cc:686] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 1 LEADER]: Becoming Leader. State: Replica: 01f34263bfdb41139579e379707442c7, State: Running, Role: LEADER
I20260620 07:05:34.944624 23404 consensus_queue.cc:237] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } }
I20260620 07:05:34.949543 22888 catalog_manager.cc:5495] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01f34263bfdb41139579e379707442c7 (127.11.81.193). New cstate: current_term: 1 leader_uuid: "01f34263bfdb41139579e379707442c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f34263bfdb41139579e379707442c7" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 38923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 } health_report { overall_health: UNKNOWN } } }
W20260620 07:05:34.956362 23246 tablet.cc:2369] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:05:34.971875 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:34.966531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.11.81.193:38923)
0620 07:05:34.966750 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:34.966755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:34.966774 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:05:34.966997 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:34.967001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:34.967103 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:34.967236 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:05:34.967653 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.968421 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.969786 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:34.970191 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:34.970282 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:34.970750 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:05:34.971329 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:05:34.971446 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:34.971575 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 07:05:34.971648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:05:34.980248 23200 raft_consensus.cc:1261] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 1 FOLLOWER]: Refusing update from remote peer 01f34263bfdb41139579e379707442c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:05:34.980257 23334 raft_consensus.cc:1261] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 1 FOLLOWER]: Refusing update from remote peer 01f34263bfdb41139579e379707442c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:05:34.981253 23409 consensus_queue.cc:1035] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7797f02a0d6a432ea30fdc206829c45d" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:05:34.981583 23404 consensus_queue.cc:1035] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc51950a38fb423c8ccc1c24667c7150" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:05:34.993511 23412 mvcc.cc:205] Tried to move back new op lower bound from 7298822696868458496 to 7298822696736940032. Current Snapshot: MvccSnapshot[applied={T|T < 7298822696868458496}]
I20260620 07:05:35.009205 22889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:05:35.009618 22889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:05:35.013367 22889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec0833143569497192d6b8618a0d47e5: Sending DeleteTablet for 3 replicas of tablet 19b8b911fd634a34a42f988bd1732486
I20260620 07:05:35.014686 23314 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b8b911fd634a34a42f988bd1732486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:05:35 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56148
I20260620 07:05:35.014686 23180 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b8b911fd634a34a42f988bd1732486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:05:35 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:34960
I20260620 07:05:35.015903 23037 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b8b911fd634a34a42f988bd1732486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:05:35 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:37178
I20260620 07:05:35.016026 23422 tablet_replica.cc:330] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: stopping tablet replica
I20260620 07:05:35.016462 23422 raft_consensus.cc:2229] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:05:35.016809 23424 tablet_replica.cc:330] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: stopping tablet replica
I20260620 07:05:35.016999 23422 raft_consensus.cc:2258] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:05:35.017341 23424 raft_consensus.cc:2229] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:05:35.017477 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 22969
I20260620 07:05:35.017746 23423 tablet_replica.cc:330] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: stopping tablet replica
I20260620 07:05:35.018069 23424 raft_consensus.cc:2258] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:05:35.018095 23423 raft_consensus.cc:2229] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:05:35.018540 23423 raft_consensus.cc:2258] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:05:35.020255 23424 ts_tablet_manager.cc:1905] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:05:35.021368 23423 ts_tablet_manager.cc:1905] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:05:35.027894 23424 ts_tablet_manager.cc:1918] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:05:35.028079 23424 log.cc:1196] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/19b8b911fd634a34a42f988bd1732486
I20260620 07:05:35.028362 23423 ts_tablet_manager.cc:1918] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:05:35.028633 23423 log.cc:1196] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/19b8b911fd634a34a42f988bd1732486
I20260620 07:05:35.028890 23422 ts_tablet_manager.cc:1905] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:05:35.028935 23424 ts_tablet_manager.cc:1939] T 19b8b911fd634a34a42f988bd1732486 P 01f34263bfdb41139579e379707442c7: Deleting consensus metadata
I20260620 07:05:35.029532 23423 ts_tablet_manager.cc:1939] T 19b8b911fd634a34a42f988bd1732486 P 7797f02a0d6a432ea30fdc206829c45d: Deleting consensus metadata
Jun 20 07:05:35 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:05:35.030828 22876 catalog_manager.cc:4841] TS 01f34263bfdb41139579e379707442c7 (127.11.81.193:38923): tablet 19b8b911fd634a34a42f988bd1732486 (table test-table [id=44125684ef784437ba978db9f3e6a58c]) successfully deleted
I20260620 07:05:35.031574 22875 catalog_manager.cc:4841] TS 7797f02a0d6a432ea30fdc206829c45d (127.11.81.195:33299): tablet 19b8b911fd634a34a42f988bd1732486 (table test-table [id=44125684ef784437ba978db9f3e6a58c]) successfully deleted
I20260620 07:05:35.034569 23422 ts_tablet_manager.cc:1918] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:05:35.034711 23422 log.cc:1196] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/19b8b911fd634a34a42f988bd1732486
I20260620 07:05:35.035102 23422 ts_tablet_manager.cc:1939] T 19b8b911fd634a34a42f988bd1732486 P fc51950a38fb423c8ccc1c24667c7150: Deleting consensus metadata
I20260620 07:05:35.037115 22873 catalog_manager.cc:4841] TS fc51950a38fb423c8ccc1c24667c7150 (127.11.81.194:36087): tablet 19b8b911fd634a34a42f988bd1732486 (table test-table [id=44125684ef784437ba978db9f3e6a58c]) successfully deleted
I20260620 07:05:35.039909 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:35.018694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.11.81.193:38923)
0620 07:05:35.018890 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:35.018897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:35.019236 (+ 339us) server_negotiation.cc:407] Connection header received
0620 07:05:35.019535 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:35.019540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:35.019624 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:35.019711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:35.021523 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.022810 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.024823 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.025498 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.025612 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:35.031576 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:35.031592 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:35.031609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:35.031664 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:35.035676 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.037199 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.037210 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.037213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.037394 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.037917 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.037922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.037925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.038419 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:35.038729 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:35.039329 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 07:05:35.039576 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:05:35.040716 23097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:05:35.136612 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 22969
I20260620 07:05:35.150907 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 23108
Jun 20 07:05:35 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:05:35.166836 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:35.151826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.11.81.194:36087)
0620 07:05:35.152004 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:35.152008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:35.152026 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:05:35.152079 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:35.152082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:35.152140 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:35.152235 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:35.153599 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.154453 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.155906 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.156355 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.156465 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:35.160571 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:35.160585 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:35.160601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:35.160639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:35.163961 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.164902 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.164911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.164914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.165046 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.165461 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.165464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.165466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.165837 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:35.166098 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:35.166404 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:05:35.166581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:05:35.167625 23240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:05:35.256773 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 23108
I20260620 07:05:35.270133 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 23251
Jun 20 07:05:35 dist-test-slave-wvz3 krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:05:35.285501 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:05:35.270952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.11.81.195:33299)
0620 07:05:35.271102 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:05:35.271107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:05:35.271124 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:05:35.271184 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:05:35.271189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:05:35.271252 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:05:35.271369 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:05:35.272608 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.273415 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.274746 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:05:35.275143 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:05:35.275247 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:05:35.279722 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:05:35.279739 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:05:35.279756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:05:35.279797 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:05:35.282556 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.283472 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.283480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.283482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.283614 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:05:35.284000 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:05:35.284003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:05:35.284005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:05:35.284392 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:05:35.284593 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 07:05:35.284958 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 07:05:35.285142 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:05:35.286160 23374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:05:35.374020 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 23251
I20260620 07:05:35.386883 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 22857
I20260620 07:05:35.388144 22919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:05:35.505441 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 22857
I20260620 07:05:35.526346 11591 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:05:35.862275 11591 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:05:35.862501 11591 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:05:36.198330 11591 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:05:36.198 UTC [22443] LOG: received smart shutdown request
2026-06-20 07:05:36.201 UTC [22443] LOG: background worker "logical replication launcher" (PID 22452) exited with exit code 1
2026-06-20 07:05:36.202 UTC [22447] LOG: shutting down
2026-06-20 07:05:36.374 UTC [22443] LOG: database system is shut down
2026-06-20T07:05:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48463 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:05:36 dist-test-slave-wvz3 krb5kdc[23438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:05:36 dist-test-slave-wvz3 krb5kdc[23438](info): set up 2 sockets
Jun 20 07:05:36 dist-test-slave-wvz3 krb5kdc[23438](info): commencing operation
krb5kdc: starting...
W20260620 07:05:37.503499 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:05:37 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:05:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangeradmin_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangeradmin_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangerlookup_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangerlookup_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
I20260620 07:05:37.819065 11591 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOWNMWE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/postgres -l logfile start
2026-06-20 07:05:40.332 UTC [23468] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:05:40.332 UTC [23468] LOG: listening on IPv4 address "127.11.81.212", port 53827
2026-06-20 07:05:40.336 UTC [23468] LOG: listening on Unix socket "/tmp/.s.PGSQL.53827"
2026-06-20 07:05:40.358 UTC [23471] LOG: database system was shut down at 2026-06-20 07:05:38 UTC
2026-06-20 07:05:40.361 UTC [23468] LOG: database system is ready to accept connections
I20260620 07:05:41.251143 11591 mini_postgres.cc:96] Postgres bound to 53827
2026-06-20 07:05:41.267 UTC [23479] FATAL: database "slave" does not exist
127.11.81.212:53827 - accepting connections
I20260620 07:05:41.268465 11591 mini_ranger.cc:161] Starting Ranger...
I20260620 07:05:41.293759 11591 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:05:41.784446 11591 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:05:41.784565 11591 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin
2026-06-20 07:05:42,065 [I] DB FLAVOR :POSTGRES
2026-06-20 07:05:42,066 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:05:42,066 [I] Checking connection..
2026-06-20 07:05:42,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:05:42,330 [I] Checking connection passed.
2026-06-20 07:05:42,331 [I] --------- Verifying version history table ---------
2026-06-20 07:05:42,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:05:42,573 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:05:42,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:05:42,827 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:05:42,827 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:05:42,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:05:43.086 UTC [23543] WARNING: there is no transaction in progress
2026-06-20 07:05:43,092 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:05:43,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:05:43,329 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:05:43,329 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:05:43,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:05:43,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:05:43,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wvz3.c.gcp-upstream.internal','N') ;"
2026-06-20 07:05:44,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:05:44,331 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:05:44,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:05:44,574 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:05:44,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:05:44,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:05:45,093 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:05:45,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:05:46.050 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.068 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.085 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.103 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.119 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.142 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.202 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.209 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.224 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.235 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.247 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.262 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.273 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.282 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.292 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46.299 UTC [23651] WARNING: there is no transaction in progress
2026-06-20 07:05:46,950 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:05:46,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:05:47,202 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:05:47,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:05:47,459 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:05:47,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:05:47,706 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:05:47,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:05:47,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:05:48,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wvz3.c.gcp-upstream.internal';"
2026-06-20 07:05:48,481 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:05:48,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:05:48,716 [I] DB_PATCHES have already been applied
I20260620 07:05:48.722237 11591 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOWNMWE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:05:48.722328 11591 mini_ranger.cc:193] Using host: 127.11.81.212
I20260620 07:05:48.735849 11591 mini_ranger.cc:227] Ranger admin URL: http://127.11.81.212:56067
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:05:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:05:49.777540 11591 mini_ranger.cc:160] Time spent starting Ranger: real 8.509s user 0.000s sys 0.039s
Jun 20, 2026 7:06:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:06:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:06:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:06:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:06:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:06:10.837280 11591 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.81.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.81.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangerkms_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.81.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/rangerkms_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.81.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.81.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.81.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/HTTP_127.11.81.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:06:10 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939170, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:06:10.999867 11591 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:06:11.019615 11591 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:06:11.912774 11591 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:06:11.912889 11591 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms
I20260620 07:06:11.912931 11591 mini_ranger_kms.cc:227] Using postgres at 127.11.81.212:53827
2026-06-20 07:06:12,245 [I] DB FLAVOR :POSTGRES
2026-06-20 07:06:12,246 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:06:12,246 [I] Checking connection
2026-06-20 07:06:12,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:06:12,476 [I] connection success
2026-06-20 07:06:12,477 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:06:12,477 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:06:12,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:06:12,705 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:06:12,706 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:06:12,706 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:06:12,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.81.212:53827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:06:12,962 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:06:13.398350 11591 mini_ranger_kms.cc:310] Created kms service
I20260620 07:06:13.608228 11591 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:06:13.651648 11591 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:06:13 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939170, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:14.036082 11591 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:06:14.036429 11591 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOWNMWE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOWNMWE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:06:14.036492 11591 mini_ranger_kms.cc:240] Using host: 127.11.81.212
I20260620 07:06:14.053434 11591 mini_ranger_kms.cc:276] Ranger KMS PID: 23831
I20260620 07:06:14.053568 11591 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.81.212:33177
Jun 20, 2026 7:06:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:06:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:06:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:06:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:06:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:06:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 07:06:16 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939176, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:16.143455 11591 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.144s user 0.083s sys 0.270s
I20260620 07:06:16.143700 11591 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:06:16.143898 11591 mini_ranger_kms.cc:401] 127.11.81.212:33177/kms/v1/keys
Jun 20 07:06:16 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939176, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
Jun 20 07:06:16 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939176, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.81.212@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
Jun 20 07:06:19 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:06:19.247370 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:46841
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:46841
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:19.571208 23882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:19.571528 23882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:19.571617 23882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:19.578909 23882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:19.579027 23882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:19.579073 23882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:19.579129 23882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:19.579175 23882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:19.589407 23882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.81.212:33177/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:46841
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:19.590706 23882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:19.592096 23882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:19.603780 23888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:19.603823 23887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:19.604259 23890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:19.604569 23882 server_base.cc:1034] running on GCE node
I20260620 07:06:19.605368 23882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:19.606827 23882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:19.608065 23882 hybrid_clock.cc:648] HybridClock initialized: now 1781939179608021 us; error 40 us; skew 500 ppm
I20260620 07:06:19.608521 23882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:19 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:19.611877 23882 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:19.613428 23882 webserver.cc:460] Webserver started at http://127.11.81.254:37809/ using document root <none> and password file <none>
I20260620 07:06:19.614113 23882 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:19.614228 23882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:19.614529 23882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:06:19 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:19.675994 23882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "cd288acb905c4fec97475ce095b3fa9d"
format_stamp: "Formatted at 2026-06-20 07:06:19 on dist-test-slave-wvz3"
server_key: "dc11ce789c2efc6d1f75eeae842c4a23"
server_key_iv: "0ab02de432342119e6d71ec7c5ef7c38"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:19.676890 23882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "cd288acb905c4fec97475ce095b3fa9d"
format_stamp: "Formatted at 2026-06-20 07:06:19 on dist-test-slave-wvz3"
server_key: "dc11ce789c2efc6d1f75eeae842c4a23"
server_key_iv: "0ab02de432342119e6d71ec7c5ef7c38"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:19.682932 23882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 07:06:19.703698 23897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:19.705472 23882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:06:19.705684 23882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "cd288acb905c4fec97475ce095b3fa9d"
format_stamp: "Formatted at 2026-06-20 07:06:19 on dist-test-slave-wvz3"
server_key: "dc11ce789c2efc6d1f75eeae842c4a23"
server_key_iv: "0ab02de432342119e6d71ec7c5ef7c38"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:19.705946 23882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:19.721539 23882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:19.725739 23882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:19.726078 23882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:19.745702 23882 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:46841
I20260620 07:06:19.745702 23948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:46841 every 8 connection(s)
I20260620 07:06:19.747164 23882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:19.750710 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 23882
I20260620 07:06:19.750922 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:06:19.750947 23949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:19.751688 11591 external_mini_cluster.cc:1428] Setting key f63be452b604d647355fc484ae066009
I20260620 07:06:19.764207 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Bootstrap starting.
Jun 20 07:06:19 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:19.768091 23949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:19.769119 23949 log.cc:824] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:19.772334 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: No bootstrap required, opened a new log
I20260620 07:06:19.775221 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:19.753696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.11.81.254:46841)
0620 07:06:19.754513 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:19.754524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:19.754574 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:06:19.756712 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:19.756738 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:19.757209 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:19.757558 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:19.759187 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:19.760595 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:19.762385 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:19.763118 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:19.763250 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:19.767860 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:19.767869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:19.767885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:19.767934 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:19.771196 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:19.771922 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:19.771930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:19.771939 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:19.772129 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:19.772476 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:19.772482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:19.772485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:19.773012 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:19.773286 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:19.773741 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 07:06:19.774205 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260620 07:06:19.780035 23949 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:19.780401 23949 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:19.780493 23949 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd288acb905c4fec97475ce095b3fa9d, State: Initialized, Role: FOLLOWER
I20260620 07:06:19.781031 23949 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:19.781237 23949 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:19.781363 23949 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:19.781525 23949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:19.783814 23949 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:19.784248 23949 leader_election.cc:302] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd288acb905c4fec97475ce095b3fa9d; no voters:
I20260620 07:06:19.784785 23949 leader_election.cc:288] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:06:19.784962 23954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:19.785568 23954 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 LEADER]: Becoming Leader. State: Replica: cd288acb905c4fec97475ce095b3fa9d, State: Running, Role: LEADER
I20260620 07:06:19.786141 23954 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:19.786684 23949 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:19.788831 23956 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd288acb905c4fec97475ce095b3fa9d. Latest consensus state: current_term: 1 leader_uuid: "cd288acb905c4fec97475ce095b3fa9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } } }
I20260620 07:06:19.789207 23956 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:19.789072 23955 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd288acb905c4fec97475ce095b3fa9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } } }
I20260620 07:06:19.789305 23955 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:19.791885 23962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:19.797907 23962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:19.798640 23949 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:06:19.815754 23962 catalog_manager.cc:1347] Generated new cluster ID: d402c4875acf4c6c94e393b672b070b7
I20260620 07:06:19.815896 23962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:19.836387 23962 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:06:19.838868 23962 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:19.854182 23962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Generated new TSK 0
I20260620 07:06:19.855420 23962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:06:20.598839 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:20 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:06:20.998896 23994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:20.999193 23994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:20.999392 23994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:21.007508 23994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:21.007633 23994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:21.007786 23994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:06:21.019419 23994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:21.021209 23994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:21.022886 23994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:21.046640 24004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:21.046617 24005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:21.055792 23994 server_base.cc:1034] running on GCE node
W20260620 07:06:21.056728 24007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:21.057905 23994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:21.060844 23994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:21.062192 23994 hybrid_clock.cc:648] HybridClock initialized: now 1781939181062083 us; error 119 us; skew 500 ppm
I20260620 07:06:21.062633 23994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:21.067070 23994 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:06:21.069347 23994 webserver.cc:460] Webserver started at http://127.11.81.193:45623/ using document root <none> and password file <none>
I20260620 07:06:21.070304 23994 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:21.070446 23994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:21.070843 23994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:21.094762 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "701eac71f6d94feaa69c2d74aed422c2"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "0b91621d1c9234eeffc2e63bd573efef"
server_key_iv: "cb49ee5bc8678cbec5783727200056a4"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:21.096043 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "701eac71f6d94feaa69c2d74aed422c2"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "0b91621d1c9234eeffc2e63bd573efef"
server_key_iv: "cb49ee5bc8678cbec5783727200056a4"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:21.103986 23994 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 07:06:21.132171 24014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:21.134202 23994 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260620 07:06:21.134441 23994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "701eac71f6d94feaa69c2d74aed422c2"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "0b91621d1c9234eeffc2e63bd573efef"
server_key_iv: "cb49ee5bc8678cbec5783727200056a4"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:21.134721 23994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:21.160181 23994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:21.165438 23994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:21.165767 23994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:21.166915 23994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:21.168612 23994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:21.168705 23994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:21.168828 23994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:21.168884 23994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:21.259562 23994 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39237
I20260620 07:06:21.260103 24126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39237 every 8 connection(s)
I20260620 07:06:21.262588 23994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:06:21.263746 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 23994
I20260620 07:06:21.263933 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:06:21.264746 11591 external_mini_cluster.cc:1428] Setting key 21bb483736b81ec4d5e8cc11ff59c5c5
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:21.305557 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:21.271689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:50415 (local address 127.11.81.254:46841)
0620 07:06:21.282756 (+ 11067us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:21.282763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:21.286470 (+ 3707us) server_negotiation.cc:407] Connection header received
0620 07:06:21.286607 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:21.286613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:21.286723 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:21.286841 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:21.288987 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:21.290121 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:21.292587 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:21.293226 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:21.293328 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:21.298037 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:21.298046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:21.298051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:21.298095 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:21.301699 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:21.302640 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:21.302645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:21.302649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:21.302800 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:21.303345 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:21.303351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:21.303355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:21.303719 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:21.303874 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:21.304924 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0620 07:06:21.305179 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10923,"thread_start_us":10798,"threads_started":1}
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:06:21.308168 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:21.270996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.193:50415)
0620 07:06:21.283446 (+ 12450us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:21.283507 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:21.286010 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:21.287173 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:21.287186 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:21.287905 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:21.288732 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:21.288764 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:21.290318 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:21.290324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:21.292401 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:21.292423 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:21.293700 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:21.293709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:21.294035 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:21.294822 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:21.294852 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:21.297679 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:21.301898 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:21.301907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:21.301928 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:21.302496 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:21.302914 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:21.302918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:21.302921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:21.303166 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:21.303956 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:21.303964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:21.304455 (+ 491us) client_negotiation.cc:769] Sending connection context
0620 07:06:21.305784 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0620 07:06:21.306383 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12079,"thread_start_us":11852,"threads_started":1}
I20260620 07:06:21.310853 24127 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:21.311252 24127 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:21.312242 24127 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:21.315191 23914 ts_manager.cc:194] Registered new tserver with Master: 701eac71f6d94feaa69c2d74aed422c2 (127.11.81.193:39237)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:06:21.317185 23914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:50415
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:06:21.359009 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939180, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
W20260620 07:06:21.839383 24133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:21.839847 24133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:21.840029 24133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:21.851778 24133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:21.851984 24133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:21.852198 24133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:06:21.867889 24133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:21.869839 24133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:21.871721 24133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:21.907868 24140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:21.911415 24141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:21.930510 24143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:21.931131 24133 server_base.cc:1034] running on GCE node
I20260620 07:06:21.932170 24133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:21.938387 24133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:21.939845 24133 hybrid_clock.cc:648] HybridClock initialized: now 1781939181939740 us; error 108 us; skew 500 ppm
I20260620 07:06:21.940310 24133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:21.944696 24133 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:06:21.957918 24133 webserver.cc:460] Webserver started at http://127.11.81.194:35935/ using document root <none> and password file <none>
I20260620 07:06:21.959008 24133 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:21.959174 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:21.959625 24133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:06:21 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:21.978183 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "78715cc914ff43b38e902182d87aa9d1"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "39b172ca3e0c49c3a8dc01853833e96d"
server_key_iv: "3b4fa966674b748ea36ca6992572be56"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:21.979123 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "78715cc914ff43b38e902182d87aa9d1"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "39b172ca3e0c49c3a8dc01853833e96d"
server_key_iv: "3b4fa966674b748ea36ca6992572be56"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:21.984954 24133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:06:22.005611 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.007639 24133 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260620 07:06:22.007824 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "78715cc914ff43b38e902182d87aa9d1"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "39b172ca3e0c49c3a8dc01853833e96d"
server_key_iv: "3b4fa966674b748ea36ca6992572be56"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:22.008057 24133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:22.027549 24133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:22.031925 24133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:22.032218 24133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:22.033217 24133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:22.034688 24133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:22.034793 24133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.034929 24133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:22.034981 24133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.070034 24133 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:34097
I20260620 07:06:22.070106 24269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:34097 every 8 connection(s)
I20260620 07:06:22.071511 24133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:06:22.073365 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24133
I20260620 07:06:22.073527 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:06:22.074060 11591 external_mini_cluster.cc:1428] Setting key 139b58e0142663e982f62baf1219c347
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939181, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:22.115639 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.074824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:49897 (local address 127.11.81.254:46841)
0620 07:06:22.087673 (+ 12849us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:22.087679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:22.087703 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:06:22.089954 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:22.089961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:22.090066 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:22.090182 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:22.092202 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.093263 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.095623 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.096289 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.096425 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.108321 (+ 11896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:22.108329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:22.108334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:22.108379 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:22.111866 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.112879 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.112885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.112888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.113049 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.113553 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.113557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.113561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.113924 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:22.114066 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:22.114964 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 07:06:22.115216 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12720,"thread_start_us":16495,"threads_started":1}
I20260620 07:06:22.116938 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.075418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.194:49897)
0620 07:06:22.086777 (+ 11359us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:22.086896 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:22.089553 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:22.090472 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:22.090485 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:22.091111 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:22.092003 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:22.092032 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.093433 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.093438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:22.095455 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:22.095475 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.096526 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.096532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:22.096805 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.097576 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:22.097605 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:22.108014 (+ 10409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:22.112056 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:22.112068 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:22.112088 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:22.112726 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:22.113171 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:22.113177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:22.113181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:22.113437 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:22.114048 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:22.114057 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:22.114533 (+ 476us) client_negotiation.cc:769] Sending connection context
0620 07:06:22.114843 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 07:06:22.115435 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11013,"thread_start_us":10672,"threads_started":1}
I20260620 07:06:22.118763 24270 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:22.119102 24270 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:22.119993 24270 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:22.121702 23914 ts_manager.cc:194] Registered new tserver with Master: 78715cc914ff43b38e902182d87aa9d1 (127.11.81.194:34097)
I20260620 07:06:22.122871 23914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:49897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:06:22.132766 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:22.321265 24127 heartbeater.cc:503] Master 127.11.81.254:46841 was elected leader, sending a full tablet report...
W20260620 07:06:22.451882 24276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:22.452188 24276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:22.452332 24276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:22.459483 24276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:22.459602 24276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:22.459753 24276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:06:22.469511 24276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:22.470866 24276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:22.472281 24276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:22.484943 24281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:22.484973 24284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:22.485101 24282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:22.486040 24276 server_base.cc:1034] running on GCE node
I20260620 07:06:22.486689 24276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:22.487723 24276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:22.488950 24276 hybrid_clock.cc:648] HybridClock initialized: now 1781939182488892 us; error 58 us; skew 500 ppm
I20260620 07:06:22.489279 24276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939182, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:22.492396 24276 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:06:22.493786 24276 webserver.cc:460] Webserver started at http://127.11.81.195:33051/ using document root <none> and password file <none>
I20260620 07:06:22.494451 24276 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:22.494565 24276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:22.494863 24276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939182, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:22.509270 24276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "1f11ffab37c34660a7c954fadbf34668"
format_stamp: "Formatted at 2026-06-20 07:06:22 on dist-test-slave-wvz3"
server_key: "d5c03ff908c7b8dde8876e08d149d225"
server_key_iv: "bca4419c3104f75356ee884bcbc5365a"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:22.510190 24276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "1f11ffab37c34660a7c954fadbf34668"
format_stamp: "Formatted at 2026-06-20 07:06:22 on dist-test-slave-wvz3"
server_key: "d5c03ff908c7b8dde8876e08d149d225"
server_key_iv: "bca4419c3104f75356ee884bcbc5365a"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:22.516503 24276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 07:06:22.531891 24291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.533640 24276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 07:06:22.533818 24276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "1f11ffab37c34660a7c954fadbf34668"
format_stamp: "Formatted at 2026-06-20 07:06:22 on dist-test-slave-wvz3"
server_key: "d5c03ff908c7b8dde8876e08d149d225"
server_key_iv: "bca4419c3104f75356ee884bcbc5365a"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:22.534039 24276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:22.573176 24276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:22.577323 24276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:22.577627 24276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:22.578599 24276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:22.579949 24276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:22.580024 24276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.580143 24276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:22.580194 24276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:22.610276 24276 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42831
I20260620 07:06:22.610309 24403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42831 every 8 connection(s)
I20260620 07:06:22.611714 24276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:06:22.614853 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24276
I20260620 07:06:22.615011 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:06:22.615500 11591 external_mini_cluster.cc:1428] Setting key ffea15d322ed92f7c2ad4422fb63f80f
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939182, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:22.632366 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.614478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:60977 (local address 127.11.81.254:46841)
0620 07:06:22.614687 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:22.614694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:22.616058 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 07:06:22.618087 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:22.618092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:22.618195 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:22.618323 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:22.619781 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.620519 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.622134 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.622598 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.622699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.626083 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:22.626089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:22.626092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:22.626128 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:22.628872 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.629854 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.629860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.629863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.630041 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.630599 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.630607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.630609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.631037 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:22.631190 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:22.631960 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 07:06:22.632145 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:06:22.633273 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.615073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.195:60977)
0620 07:06:22.615880 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:22.615931 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:22.617833 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:22.618563 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:22.618572 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:22.618989 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:22.619615 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:22.619633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.620668 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.620672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:22.621995 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:22.622006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.622806 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.622810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:22.623020 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.623647 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:22.623674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:22.625842 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:22.629017 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:22.629026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:22.629044 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:22.629671 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:22.630156 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:22.630161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:22.630164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:22.630471 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:22.631164 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:22.631171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:22.631583 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 07:06:22.631809 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 07:06:22.632197 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":342,"threads_started":1}
I20260620 07:06:22.634513 24404 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:22.634805 24404 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:22.635546 24404 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:22.636956 23914 ts_manager.cc:194] Registered new tserver with Master: 1f11ffab37c34660a7c954fadbf34668 (127.11.81.195:42831)
I20260620 07:06:22.637784 23914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:60977
I20260620 07:06:22.643482 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:06:22.645138 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 23994
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:06:22.662432 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.646179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.11.81.193:39237)
0620 07:06:22.646815 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:22.646822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:22.646848 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:06:22.646904 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:22.646910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:22.647125 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:22.647254 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:22.648764 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.649815 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.651390 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.652022 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.652129 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.656638 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:22.656652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:22.656667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:22.656723 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:22.659543 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.660285 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.660292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.660295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.660454 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.660993 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.660997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.660999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.661361 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:22.661604 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:22.661937 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 07:06:22.662142 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":341,"threads_started":1}
I20260620 07:06:22.663759 24122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:22.745538 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 23994
I20260620 07:06:22.762575 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24133
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:06:22.778756 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.763612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.11.81.194:34097)
0620 07:06:22.764248 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:22.764255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:22.764281 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:06:22.764349 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:22.764355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:22.764587 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:22.764713 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:22.766088 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.767160 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.768582 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.769205 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.769319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.773456 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:22.773468 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:22.773484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:22.773519 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:22.776177 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.776842 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.776849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.776851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.776986 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.777367 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.777370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.777372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.777713 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:22.777922 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:22.778204 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 07:06:22.778410 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":368,"threads_started":1}
I20260620 07:06:22.780037 24265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:22.860913 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24133
I20260620 07:06:22.873142 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24276
Jun 20 07:06:22 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:06:22.889810 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:22.874128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.11.81.195:42831)
0620 07:06:22.874765 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:22.874772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:22.874802 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:06:22.874864 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:22.874872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:22.875082 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:22.875208 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:22.876527 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.877537 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.878766 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:22.879461 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:22.879557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:22.884003 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:22.884014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:22.884030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:22.884081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:22.886789 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.887683 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.887691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.887693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.887826 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:22.888227 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:22.888231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:22.888233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:22.888597 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:22.888823 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:22.889167 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:06:22.889460 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":368,"threads_started":1}
I20260620 07:06:22.891348 24399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:22.972623 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24276
I20260620 07:06:22.984951 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 23882
I20260620 07:06:22.986227 23944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:23.092149 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 23882
I20260620 07:06:23.112793 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:46841
--webserver_interface=127.11.81.254
--webserver_port=37809
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:46841
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:23.444585 24415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:23.444911 24415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:23.445008 24415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:23.452615 24415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:23.452750 24415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:23.452813 24415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:23.452912 24415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:23.452976 24415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:23.463406 24415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:46841
--ranger_config_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=37809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:23.464737 24415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:23.466089 24415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:23.478268 24421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:23.478312 24423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:23.478241 24420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:23.479740 24415 server_base.cc:1034] running on GCE node
I20260620 07:06:23.480785 24415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:23.482681 24415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:23.483899 24415 hybrid_clock.cc:648] HybridClock initialized: now 1781939183483824 us; error 67 us; skew 500 ppm
I20260620 07:06:23.484338 24415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:23 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:23.492681 24415 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:23.494267 24415 webserver.cc:460] Webserver started at http://127.11.81.254:37809/ using document root <none> and password file <none>
I20260620 07:06:23.494933 24415 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:23.495050 24415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:06:23 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939183, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:23.547832 24415 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:06:23.551201 24431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:23.552870 24415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:06:23.553083 24415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "cd288acb905c4fec97475ce095b3fa9d"
format_stamp: "Formatted at 2026-06-20 07:06:19 on dist-test-slave-wvz3"
server_key: "dc11ce789c2efc6d1f75eeae842c4a23"
server_key_iv: "0ab02de432342119e6d71ec7c5ef7c38"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:23.554047 24415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:23.576457 24415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:23.580400 24415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:23.580679 24415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:23.599263 24415 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:46841
I20260620 07:06:23.599320 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:46841 every 8 connection(s)
I20260620 07:06:23.600785 24415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:23.605894 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24415
I20260620 07:06:23.606693 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:39237
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:46841
--webserver_port=45623
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:23.608354 24483 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:06:23.611886 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Bootstrap starting.
I20260620 07:06:23.630904 24483 log.cc:824] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:23.639991 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:06:23.640506 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Bootstrap complete.
I20260620 07:06:23.649397 24483 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:23.650288 24483 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd288acb905c4fec97475ce095b3fa9d, State: Initialized, Role: FOLLOWER
I20260620 07:06:23.650947 24483 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:23.651167 24483 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:23.651304 24483 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:23.651484 24483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:06:23.655479 24483 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:23.656006 24483 leader_election.cc:302] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd288acb905c4fec97475ce095b3fa9d; no voters:
I20260620 07:06:23.656620 24483 leader_election.cc:288] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:06:23.656764 24487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:06:23.657823 24487 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [term 2 LEADER]: Becoming Leader. State: Replica: cd288acb905c4fec97475ce095b3fa9d, State: Running, Role: LEADER
I20260620 07:06:23.658476 24487 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } }
I20260620 07:06:23.658910 24483 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:23.660948 24489 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd288acb905c4fec97475ce095b3fa9d. Latest consensus state: current_term: 2 leader_uuid: "cd288acb905c4fec97475ce095b3fa9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } } }
I20260620 07:06:23.661324 24489 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:23.661955 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd288acb905c4fec97475ce095b3fa9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd288acb905c4fec97475ce095b3fa9d" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 46841 } } }
I20260620 07:06:23.662253 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:23.672312 24493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:23.678508 24493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:23.680020 24493 catalog_manager.cc:1259] Loaded cluster ID: d402c4875acf4c6c94e393b672b070b7
I20260620 07:06:23.680200 24493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:23.684255 24493 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:23.685854 24493 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Loaded TSK: 0
I20260620 07:06:23.687645 24493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:06:23.692755 24483 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 07:06:24.010079 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:24.010754 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:24.010975 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:24.021222 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:24.021467 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:24.021672 24485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:06:24.034595 24485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=45623
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:24.036046 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:24.037891 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:24.061075 24520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:24.061767 24521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:24.066607 24523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:24.066841 24485 server_base.cc:1034] running on GCE node
I20260620 07:06:24.067724 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:24.069211 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:24.072629 24485 hybrid_clock.cc:648] HybridClock initialized: now 1781939184072555 us; error 59 us; skew 500 ppm
I20260620 07:06:24.073146 24485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:24.077881 24485 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:06:24.101464 24485 webserver.cc:460] Webserver started at http://127.11.81.193:45623/ using document root <none> and password file <none>
I20260620 07:06:24.102555 24485 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:24.102737 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:24.137344 24485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:06:24.143699 24531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.145653 24485 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260620 07:06:24.145843 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "701eac71f6d94feaa69c2d74aed422c2"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "0b91621d1c9234eeffc2e63bd573efef"
server_key_iv: "cb49ee5bc8678cbec5783727200056a4"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:24.146850 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:24.182229 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:24.188066 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:24.188486 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:24.206524 24485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:24.209862 24485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:24.209996 24485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.210160 24485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:24.210232 24485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.273604 24485 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39237
I20260620 07:06:24.273653 24644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39237 every 8 connection(s)
I20260620 07:06:24.275178 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:06:24.285759 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24485
I20260620 07:06:24.286620 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:34097
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:46841
--webserver_port=35935
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:24.317950 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:24.278890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.193:37669)
0620 07:06:24.290240 (+ 11350us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:24.290309 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:24.292904 (+ 2595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:24.296202 (+ 3298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:24.296211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:24.296868 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:24.297697 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:24.297727 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.300197 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.300203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:24.302314 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:24.302330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.304251 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.304260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:24.304565 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:24.305367 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:24.305399 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:24.307941 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:24.312466 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:24.312476 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:24.312494 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:24.313125 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:24.313905 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:24.313911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:24.313914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:24.314190 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:24.315325 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:24.315334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:24.315754 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 07:06:24.316043 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 07:06:24.316616 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10941,"thread_start_us":14412,"threads_started":1}
I20260620 07:06:24.317978 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:24.278840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:37669 (local address 127.11.81.254:46841)
0620 07:06:24.293852 (+ 15012us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:24.293875 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:24.293926 (+ 51us) server_negotiation.cc:407] Connection header received
0620 07:06:24.294367 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:24.294385 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:24.294897 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:24.295343 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:24.297964 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.299990 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.302495 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.303645 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.303854 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:24.308272 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:24.308283 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:24.308305 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:24.308353 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:24.312242 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:24.313309 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:24.313319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:24.313329 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:24.313525 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:24.314359 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:24.314365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:24.314368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:24.315117 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:24.315378 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:24.316309 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 07:06:24.316735 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14714,"thread_start_us":16791,"threads_started":1}
I20260620 07:06:24.322160 24645 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:24.322522 24645 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:24.323442 24645 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
W20260620 07:06:24.326917 24645 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:46841 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:06:24.327164 24645 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:24.327518 24645 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:24.328961 24645 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:24.329340 24645 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
W20260620 07:06:24.330502 24645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:06:24.671782 24649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:24.672098 24649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:24.672254 24649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:24.682016 24649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:24.682178 24649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:24.682395 24649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:06:24.694206 24649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:34097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=35935
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:24.695973 24649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:24.697655 24649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:24.713886 24661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:24.713903 24662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:24.714627 24664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:24.715087 24649 server_base.cc:1034] running on GCE node
I20260620 07:06:24.715868 24649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:24.717142 24649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:24.718405 24649 hybrid_clock.cc:648] HybridClock initialized: now 1781939184718339 us; error 46 us; skew 500 ppm
I20260620 07:06:24.718806 24649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:24.722841 24649 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:06:24.724720 24649 webserver.cc:460] Webserver started at http://127.11.81.194:35935/ using document root <none> and password file <none>
I20260620 07:06:24.725654 24649 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:24.725812 24649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:24.757782 24649 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.001s sys 0.006s
I20260620 07:06:24.761801 24674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.763825 24649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:06:24.764007 24649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "78715cc914ff43b38e902182d87aa9d1"
format_stamp: "Formatted at 2026-06-20 07:06:21 on dist-test-slave-wvz3"
server_key: "39b172ca3e0c49c3a8dc01853833e96d"
server_key_iv: "3b4fa966674b748ea36ca6992572be56"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:24.765440 24649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:24.783157 24649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:24.788549 24649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:24.788882 24649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:24.790153 24649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:24.792136 24649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:24.792259 24649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.792377 24649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:24.792457 24649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:24.902412 24649 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:34097
I20260620 07:06:24.902500 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:34097 every 8 connection(s)
I20260620 07:06:24.904357 24649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:06:24.909778 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24649
I20260620 07:06:24.910631 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.81.212:33177/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:42831
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:46841
--webserver_port=33051
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:24.941483 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:24.907631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60363 (local address 127.11.81.254:46841)
0620 07:06:24.921304 (+ 13673us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:24.921311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:24.921339 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:06:24.921680 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:24.921687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:24.921782 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:24.921914 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:24.924421 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.925628 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.928187 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.928864 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.928970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:24.933720 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:24.933728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:24.933733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:24.933774 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:24.937400 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:24.938433 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:24.938439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:24.938443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:24.938596 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:24.939161 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:24.939166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:24.939169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:24.939678 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:24.939831 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:24.940839 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 07:06:24.941086 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13529,"thread_start_us":23402,"threads_started":1}
Jun 20 07:06:24 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:24.944235 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:24.908126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.194:60363)
0620 07:06:24.918757 (+ 10631us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:24.918812 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:24.921340 (+ 2528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:24.922429 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:24.922443 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:24.923118 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:24.924144 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:24.924175 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.925840 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.925845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:24.927991 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:24.928012 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:24.929381 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:24.929401 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:24.929712 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:24.930561 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:24.930619 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:24.933431 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:24.937617 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:24.937626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:24.937646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:24.938267 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:24.938748 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:24.938752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:24.938756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:24.939026 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:24.939928 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:24.939937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:24.940397 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 07:06:24.942087 (+ 1690us) client_negotiation.cc:240] Negotiation successful
0620 07:06:24.942709 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10238,"thread_start_us":10352,"threads_started":1}
I20260620 07:06:24.946264 24789 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:24.946628 24789 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:24.947623 24789 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:24.950019 24448 ts_manager.cc:194] Registered new tserver with Master: 78715cc914ff43b38e902182d87aa9d1 (127.11.81.194:34097)
I20260620 07:06:24.952356 24448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:60363
I20260620 07:06:25.330667 24645 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:25.331112 24645 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:25.332541 24448 ts_manager.cc:194] Registered new tserver with Master: 701eac71f6d94feaa69c2d74aed422c2 (127.11.81.193:39237)
I20260620 07:06:25.333684 24448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:37669
Jun 20 07:06:25 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
W20260620 07:06:25.378736 24793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:25.379249 24793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:25.379459 24793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:25.390811 24793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:25.390969 24793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:25.391211 24793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:06:25.406677 24793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.81.212:33177/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:42831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=33051
--tserver_master_addrs=127.11.81.254:46841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:25.408385 24793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:25.410123 24793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:25.431046 24799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:25.433764 24800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:25.435050 24793 server_base.cc:1034] running on GCE node
W20260620 07:06:25.435742 24802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:25.436570 24793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:25.437772 24793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:25.439064 24793 hybrid_clock.cc:648] HybridClock initialized: now 1781939185439000 us; error 49 us; skew 500 ppm
I20260620 07:06:25.439487 24793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:25 dist-test-slave-wvz3 krb5kdc[23438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939185, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:25.443658 24793 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:06:25.446977 24793 webserver.cc:460] Webserver started at http://127.11.81.195:33051/ using document root <none> and password file <none>
I20260620 07:06:25.447932 24793 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:25.448091 24793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:06:25 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939185, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:25.492239 24793 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.011s sys 0.009s
I20260620 07:06:25.496207 24810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:25.498198 24793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:06:25.498461 24793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "1f11ffab37c34660a7c954fadbf34668"
format_stamp: "Formatted at 2026-06-20 07:06:22 on dist-test-slave-wvz3"
server_key: "d5c03ff908c7b8dde8876e08d149d225"
server_key_iv: "bca4419c3104f75356ee884bcbc5365a"
server_key_version: "kuduclusterkey@0"
I20260620 07:06:25.500066 24793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:25.547431 24793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:25.557587 24793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:25.558050 24793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:25.559538 24793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:25.561544 24793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:25.561693 24793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 07:06:25.561832 24793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:25.561954 24793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:06:25 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939184, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for HTTP/127.11.81.212@KRBTEST.COM
I20260620 07:06:25.825320 24793 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:42831
I20260620 07:06:25.825408 24923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:42831 every 8 connection(s)
I20260620 07:06:25.827392 24793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:06:25.828513 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 24793
Jun 20 07:06:25 dist-test-slave-wvz3 krb5kdc[23438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939185, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:25.870170 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.829724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.11.81.254:46841)
0620 07:06:25.835744 (+ 6020us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.835751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.847090 (+ 11339us) server_negotiation.cc:407] Connection header received
0620 07:06:25.847362 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.847368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.847460 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.847573 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:25.850078 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.851443 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.855991 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.857104 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.857241 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.861891 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:25.861900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:25.861905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:25.861955 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:25.865701 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.866696 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.866702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.866706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.866867 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.867424 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.867429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.867432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.867814 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:25.868112 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.869094 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.869469 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":5849,"thread_start_us":6307,"threads_started":1}
I20260620 07:06:25.875514 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.838892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:52655 (local address 127.11.81.254:46841)
0620 07:06:25.845598 (+ 6706us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.845605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.845625 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:06:25.845694 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.845699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.845790 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.845894 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:25.849713 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.851901 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.854447 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.855095 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.855214 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.867943 (+ 12729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:25.867951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:25.867956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:25.867997 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:25.871497 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.872514 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.872519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.872523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.872658 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.873229 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.873234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.873238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.873588 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:25.873725 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.874843 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.875079 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":6566,"thread_start_us":6416,"threads_started":1}
I20260620 07:06:25.876543 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.838273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:46841 (local address 127.11.81.195:52655)
0620 07:06:25.842265 (+ 3992us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:25.842328 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:25.844795 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:25.847707 (+ 2912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:25.847722 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:25.848503 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:25.849470 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.849498 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.852118 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.852124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.854248 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.854264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.855589 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.855597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.855892 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.857527 (+ 1635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:25.857559 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:25.867577 (+ 10018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:25.871693 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:25.871705 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:25.871727 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:25.872369 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:25.872794 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:25.872799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:25.872803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:25.873095 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:25.873796 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:25.873805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:25.874278 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 07:06:25.874573 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 07:06:25.875145 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3642,"thread_start_us":3366,"threads_started":1}
I20260620 07:06:25.878738 24924 heartbeater.cc:347] Connected to a master server at 127.11.81.254:46841
I20260620 07:06:25.879065 24924 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:25.879930 24924 heartbeater.cc:511] Master 127.11.81.254:46841 requested a full tablet report, sending...
I20260620 07:06:25.881520 24448 ts_manager.cc:194] Registered new tserver with Master: 1f11ffab37c34660a7c954fadbf34668 (127.11.81.195:42831)
I20260620 07:06:25.882643 24448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:52655
I20260620 07:06:25.887629 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:06:25.904318 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.890790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.11.81.254:46841)
0620 07:06:25.890984 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.890991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.891550 (+ 559us) server_negotiation.cc:407] Connection header received
0620 07:06:25.891720 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.891725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.891817 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.891909 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:25.893483 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.894206 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.895609 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.896101 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.896214 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.898852 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:25.898858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:25.898862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:25.898896 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:25.901575 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.902320 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.902326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.902329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.902474 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:25.902830 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:25.902835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:25.902838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:25.903133 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:25.903396 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.903747 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.903995 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:06:25.911425 24448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:06:25.922740 24448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:06:25.957576 24789 heartbeater.cc:503] Master 127.11.81.254:46841 was elected leader, sending a full tablet report...
I20260620 07:06:25.959939 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.945959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:34097 (local address 127.0.0.1:53566)
0620 07:06:25.946882 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:25.946897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:25.947006 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:25.949604 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:25.949612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:25.949645 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:25.950026 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.950038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.953272 (+ 3234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.953279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.955399 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.955413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.957980 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.957990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.958250 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.958324 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 07:06:25.958462 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 07:06:25.959465 (+ 1003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":24,"thread_start_us":1742,"threads_started":1}
I20260620 07:06:25.960291 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.946188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.11.81.194:34097)
0620 07:06:25.947505 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.947513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.947539 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:06:25.947645 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.948738 (+ 1093us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.949112 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.949323 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:25.950234 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.953092 (+ 2858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.955768 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.957168 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.957305 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.957555 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.958636 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.958804 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":984,"threads_started":1}
I20260620 07:06:25.963158 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.946952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39382)
0620 07:06:25.955048 (+ 8096us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:25.955069 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:25.955184 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:25.956513 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:25.956517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:25.956540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:25.956913 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.956923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.958726 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.958731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.960808 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.960823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.962583 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.962588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.962714 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.962738 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:06:25.962820 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:06:25.962926 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7952,"thread_start_us":7806,"threads_started":1}
I20260620 07:06:25.963404 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.947505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.11.81.193:39237)
0620 07:06:25.955706 (+ 8201us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.955712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.955738 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:06:25.955834 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.955841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.956099 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.956284 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:25.957091 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.958579 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.961003 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.962353 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.962491 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.962704 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.962890 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.963045 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8050,"thread_start_us":7902,"threads_started":1}
I20260620 07:06:25.964236 24722 tablet_service.cc:1467] Processing CreateTablet for tablet c7b53378e4584c9b9228086bb87986f3 (DEFAULT_TABLE table=test-table [id=b2e6779b8b234fa8946d27bc746621bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:25.965457 24722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b53378e4584c9b9228086bb87986f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:25.966406 24579 tablet_service.cc:1467] Processing CreateTablet for tablet c7b53378e4584c9b9228086bb87986f3 (DEFAULT_TABLE table=test-table [id=b2e6779b8b234fa8946d27bc746621bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:25.967842 24579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b53378e4584c9b9228086bb87986f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:25.968523 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.945315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42831 (local address 127.0.0.1:42792)
0620 07:06:25.946191 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:25.946234 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:25.946398 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:25.954344 (+ 7946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:25.954350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:25.954378 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:25.954782 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.954798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.956810 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.956815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.959612 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:25.959626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.967696 (+ 8070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.967704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:25.967905 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.967941 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:06:25.968010 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:06:25.968155 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":19,"thread_start_us":784,"threads_started":1}
I20260620 07:06:25.969584 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:25.945860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.11.81.195:42831)
0620 07:06:25.953701 (+ 7841us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:25.953710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:25.953745 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:06:25.953847 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:25.953856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:25.954132 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:25.954331 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:25.955051 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.956623 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.959765 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:25.967424 (+ 7659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:25.968774 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:25.969024 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:25.969094 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:06:25.969261 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7657,"thread_start_us":7493,"threads_started":1}
I20260620 07:06:25.973393 24858 tablet_service.cc:1467] Processing CreateTablet for tablet c7b53378e4584c9b9228086bb87986f3 (DEFAULT_TABLE table=test-table [id=b2e6779b8b234fa8946d27bc746621bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:25.975679 24858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b53378e4584c9b9228086bb87986f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:25.978605 24952 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Bootstrap starting.
I20260620 07:06:25.981570 24952 tablet_bootstrap.cc:654] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:25.982599 24952 log.cc:824] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:25.983237 24953 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Bootstrap starting.
I20260620 07:06:25.986225 24952 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: No bootstrap required, opened a new log
I20260620 07:06:25.986501 24952 ts_tablet_manager.cc:1397] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:06:25.987082 24953 tablet_bootstrap.cc:654] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:25.988384 24953 log.cc:824] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:25.991740 24953 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: No bootstrap required, opened a new log
I20260620 07:06:25.992125 24953 ts_tablet_manager.cc:1397] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 07:06:25.992465 24956 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Bootstrap starting.
I20260620 07:06:25.995710 24952 raft_consensus.cc:348] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:25.996186 24952 raft_consensus.cc:374] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:25.996311 24952 raft_consensus.cc:729] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78715cc914ff43b38e902182d87aa9d1, State: Initialized, Role: FOLLOWER
I20260620 07:06:25.996979 24956 tablet_bootstrap.cc:654] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:25.997154 24952 consensus_queue.cc:260] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:25.998901 24952 ts_tablet_manager.cc:1428] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 07:06:25.999668 24956 log.cc:824] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:26.000636 24953 raft_consensus.cc:348] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.001076 24953 raft_consensus.cc:374] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:26.001173 24953 raft_consensus.cc:729] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701eac71f6d94feaa69c2d74aed422c2, State: Initialized, Role: FOLLOWER
I20260620 07:06:26.001816 24953 consensus_queue.cc:260] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.002722 24956 tablet_bootstrap.cc:492] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: No bootstrap required, opened a new log
I20260620 07:06:26.003067 24956 ts_tablet_manager.cc:1397] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 07:06:26.003803 24645 heartbeater.cc:503] Master 127.11.81.254:46841 was elected leader, sending a full tablet report...
I20260620 07:06:26.004523 24953 ts_tablet_manager.cc:1428] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 07:06:26.012352 24956 raft_consensus.cc:348] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.012887 24956 raft_consensus.cc:374] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:26.013005 24956 raft_consensus.cc:729] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f11ffab37c34660a7c954fadbf34668, State: Initialized, Role: FOLLOWER
I20260620 07:06:26.013636 24956 consensus_queue.cc:260] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.014734 24924 heartbeater.cc:503] Master 127.11.81.254:46841 was elected leader, sending a full tablet report...
I20260620 07:06:26.015213 24956 ts_tablet_manager.cc:1428] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260620 07:06:26.027891 24646 tablet.cc:2369] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:06:26.068540 24957 raft_consensus.cc:482] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:06:26.068857 24957 raft_consensus.cc:504] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.070508 24957 leader_election.cc:288] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f11ffab37c34660a7c954fadbf34668 (127.11.81.195:42831), 701eac71f6d94feaa69c2d74aed422c2 (127.11.81.193:39237)
I20260620 07:06:26.079114 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.071999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:33771 (local address 127.11.81.193:39237)
0620 07:06:26.072198 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.072205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.072440 (+ 235us) server_negotiation.cc:407] Connection header received
0620 07:06:26.072610 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.072616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.072691 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.072792 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:26.073339 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.074399 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.076763 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.078021 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.078154 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.078216 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.078702 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.078836 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:06:26.079089 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.070955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:58069 (local address 127.11.81.195:42831)
0620 07:06:26.071117 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.071121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.071816 (+ 695us) server_negotiation.cc:407] Connection header received
0620 07:06:26.072080 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.072087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.072171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.072241 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:26.073079 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.074145 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.076757 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.077971 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.078135 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.078210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.078701 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.078836 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:06:26.079245 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.070913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:42831 (local address 127.11.81.194:58069)
0620 07:06:26.071683 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.071702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:26.071864 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:26.072463 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:26.072468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:26.072497 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:26.072896 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:26.072910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.074273 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.074277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:26.076574 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:26.076587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.078252 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.078257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:26.078442 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.078554 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 07:06:26.078638 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:06:26.078810 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":76,"thread_start_us":429,"threads_started":1}
I20260620 07:06:26.079698 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.071834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.11.81.194:33771)
0620 07:06:26.072338 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.072352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:26.072475 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:26.072800 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:26.072802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:26.072823 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:26.073194 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:26.073203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.074514 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.074523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:26.076572 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:26.076587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.078229 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.078240 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:26.078478 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.078551 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 07:06:26.078638 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:06:26.078809 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":74,"thread_start_us":366,"threads_started":1}
I20260620 07:06:26.080302 24878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b53378e4584c9b9228086bb87986f3" candidate_uuid: "78715cc914ff43b38e902182d87aa9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f11ffab37c34660a7c954fadbf34668" is_pre_election: true
I20260620 07:06:26.080507 24599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b53378e4584c9b9228086bb87986f3" candidate_uuid: "78715cc914ff43b38e902182d87aa9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701eac71f6d94feaa69c2d74aed422c2" is_pre_election: true
I20260620 07:06:26.080817 24878 raft_consensus.cc:2454] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78715cc914ff43b38e902182d87aa9d1 in term 0.
I20260620 07:06:26.080955 24599 raft_consensus.cc:2454] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78715cc914ff43b38e902182d87aa9d1 in term 0.
I20260620 07:06:26.081440 24676 leader_election.cc:302] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f11ffab37c34660a7c954fadbf34668, 78715cc914ff43b38e902182d87aa9d1; no voters:
I20260620 07:06:26.081785 24957 raft_consensus.cc:2781] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:06:26.081939 24957 raft_consensus.cc:482] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:06:26.082047 24957 raft_consensus.cc:3037] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:26.084219 24957 raft_consensus.cc:504] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.084756 24957 leader_election.cc:288] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [CANDIDATE]: Term 1 election: Requested vote from peers 1f11ffab37c34660a7c954fadbf34668 (127.11.81.195:42831), 701eac71f6d94feaa69c2d74aed422c2 (127.11.81.193:39237)
I20260620 07:06:26.085250 24599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b53378e4584c9b9228086bb87986f3" candidate_uuid: "78715cc914ff43b38e902182d87aa9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701eac71f6d94feaa69c2d74aed422c2"
I20260620 07:06:26.085255 24878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b53378e4584c9b9228086bb87986f3" candidate_uuid: "78715cc914ff43b38e902182d87aa9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f11ffab37c34660a7c954fadbf34668"
I20260620 07:06:26.085434 24599 raft_consensus.cc:3037] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:26.085459 24878 raft_consensus.cc:3037] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:26.087757 24599 raft_consensus.cc:2454] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78715cc914ff43b38e902182d87aa9d1 in term 1.
I20260620 07:06:26.087808 24878 raft_consensus.cc:2454] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78715cc914ff43b38e902182d87aa9d1 in term 1.
I20260620 07:06:26.088140 24676 leader_election.cc:302] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701eac71f6d94feaa69c2d74aed422c2, 78715cc914ff43b38e902182d87aa9d1; no voters:
I20260620 07:06:26.088387 24957 raft_consensus.cc:2781] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:26.088915 24957 raft_consensus.cc:686] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 78715cc914ff43b38e902182d87aa9d1, State: Running, Role: LEADER
W20260620 07:06:26.089177 24925 tablet.cc:2369] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:06:26.089483 24957 consensus_queue.cc:237] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } }
I20260620 07:06:26.093947 24447 catalog_manager.cc:5495] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78715cc914ff43b38e902182d87aa9d1 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "78715cc914ff43b38e902182d87aa9d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78715cc914ff43b38e902182d87aa9d1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 34097 } health_report { overall_health: HEALTHY } } }
I20260620 07:06:26.111151 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.105829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.11.81.194:34097)
0620 07:06:26.106045 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.106049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.106085 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:06:26.106169 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.106172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.106265 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.106381 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:06:26.106712 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.107504 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.108916 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.109374 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.109463 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.109952 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:06:26.110492 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:06:26.110611 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.110753 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.110823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:06:26.119621 24599 raft_consensus.cc:1261] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 1 FOLLOWER]: Refusing update from remote peer 78715cc914ff43b38e902182d87aa9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:26.119692 24878 raft_consensus.cc:1261] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 1 FOLLOWER]: Refusing update from remote peer 78715cc914ff43b38e902182d87aa9d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:26.120574 24963 consensus_queue.cc:1035] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "701eac71f6d94feaa69c2d74aed422c2" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:26.120994 24957 consensus_queue.cc:1035] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f11ffab37c34660a7c954fadbf34668" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:26.132368 24968 mvcc.cc:205] Tried to move back new op lower bound from 7298822906336043008 to 7298822906226143232. Current Snapshot: MvccSnapshot[applied={T|T < 7298822906336043008}]
I20260620 07:06:26.132475 24966 mvcc.cc:205] Tried to move back new op lower bound from 7298822906336043008 to 7298822906226143232. Current Snapshot: MvccSnapshot[applied={T|T < 7298822906336043008}]
I20260620 07:06:26.147653 24447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:26.147907 24447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:26.152105 24447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd288acb905c4fec97475ce095b3fa9d: Sending DeleteTablet for 3 replicas of tablet c7b53378e4584c9b9228086bb87986f3
I20260620 07:06:26.153682 24722 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b53378e4584c9b9228086bb87986f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:26 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:53566
I20260620 07:06:26.153677 24858 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b53378e4584c9b9228086bb87986f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:26 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:42792
I20260620 07:06:26.154178 24579 tablet_service.cc:1514] Processing DeleteTablet for tablet c7b53378e4584c9b9228086bb87986f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:26 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:39382
I20260620 07:06:26.154870 24977 tablet_replica.cc:330] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: stopping tablet replica
I20260620 07:06:26.155453 24977 raft_consensus.cc:2229] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:06:26.155635 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24485
I20260620 07:06:26.156769 24977 raft_consensus.cc:2258] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:26.157651 24976 tablet_replica.cc:330] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: stopping tablet replica
I20260620 07:06:26.161303 24976 raft_consensus.cc:2229] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:26.161931 24976 raft_consensus.cc:2258] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:26.161932 24978 tablet_replica.cc:330] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: stopping tablet replica
I20260620 07:06:26.162451 24978 raft_consensus.cc:2229] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:26.162905 24978 raft_consensus.cc:2258] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:26.160652 24977 ts_tablet_manager.cc:1905] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:26.166206 24978 ts_tablet_manager.cc:1905] T c7b53378e4584c9b9228086bb87986f3 P 701eac71f6d94feaa69c2d74aed422c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:26.166920 24976 ts_tablet_manager.cc:1905] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:26.174557 24976 ts_tablet_manager.cc:1918] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:26.174813 24976 log.cc:1196] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/c7b53378e4584c9b9228086bb87986f3
I20260620 07:06:26.175577 24976 ts_tablet_manager.cc:1939] T c7b53378e4584c9b9228086bb87986f3 P 1f11ffab37c34660a7c954fadbf34668: Deleting consensus metadata
I20260620 07:06:26.176128 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.156839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.11.81.193:39237)
0620 07:06:26.157023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.157029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.158634 (+ 1605us) server_negotiation.cc:407] Connection header received
0620 07:06:26.158831 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.158837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.158939 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.159040 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:26.161275 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.162416 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.164714 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.165321 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.165450 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.168932 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:26.168952 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:26.168969 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:26.169014 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:26.172861 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.173844 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.173855 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.173859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.174032 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.174441 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.174446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.174449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.174915 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:26.175217 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.175546 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.175751 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:06:26.176849 24640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:26.177789 24433 catalog_manager.cc:4841] TS 1f11ffab37c34660a7c954fadbf34668 (127.11.81.195:42831): tablet c7b53378e4584c9b9228086bb87986f3 (table test-table [id=b2e6779b8b234fa8946d27bc746621bf]) successfully deleted
I20260620 07:06:26.178994 24977 ts_tablet_manager.cc:1918] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:26.179207 24977 log.cc:1196] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/c7b53378e4584c9b9228086bb87986f3
I20260620 07:06:26.179869 24977 ts_tablet_manager.cc:1939] T c7b53378e4584c9b9228086bb87986f3 P 78715cc914ff43b38e902182d87aa9d1: Deleting consensus metadata
I20260620 07:06:26.181975 24434 catalog_manager.cc:4841] TS 78715cc914ff43b38e902182d87aa9d1 (127.11.81.194:34097): tablet c7b53378e4584c9b9228086bb87986f3 (table test-table [id=b2e6779b8b234fa8946d27bc746621bf]) successfully deleted
I20260620 07:06:26.308986 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24485
I20260620 07:06:26.324703 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24649
W20260620 07:06:26.324779 24433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:39237 (error 108)
I20260620 07:06:26.327955 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.327494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39410)
0620 07:06:26.327663 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.327781 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:06:26.328374 24433 catalog_manager.cc:4584] TS 701eac71f6d94feaa69c2d74aed422c2 (127.11.81.193:39237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7b53378e4584c9b9228086bb87986f3: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
I20260620 07:06:26.340121 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.325869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.11.81.194:34097)
0620 07:06:26.326052 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.326058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.326078 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:06:26.326141 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.326146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.326239 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.326343 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:26.328240 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.329246 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.330612 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.331123 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.331254 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.333934 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:26.333949 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:26.333964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:26.334001 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:26.337265 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.338023 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.338030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.338032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.338184 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.338557 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.338560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.338563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.338944 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:26.339279 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.339539 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.339773 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:06:26.340843 24784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:26.382301 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.381772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39418)
0620 07:06:26.382039 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.382157 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 07:06:26.383440 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.382983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39420)
0620 07:06:26.383147 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.383240 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:06:26.443979 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.443474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39426)
0620 07:06:26.443705 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.443800 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 07:06:26.444945 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.444639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39442)
0620 07:06:26.444757 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.444826 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:06:26.451311 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24649
I20260620 07:06:26.466370 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24793
I20260620 07:06:26.480023 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.467095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.11.81.195:42831)
0620 07:06:26.467262 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:26.467268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:26.467353 (+ 85us) server_negotiation.cc:407] Connection header received
0620 07:06:26.467560 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:26.467564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:26.467625 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:26.467729 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:26.469037 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.469748 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.471011 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:26.471643 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:26.471756 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:26.474308 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:26.474324 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:26.474340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:26.474375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:26.477189 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.477931 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.477940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.477942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.478092 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:26.478473 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:26.478478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:26.478481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:26.478836 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:26.479140 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:26.479503 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 07:06:26.479702 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:06:26.480711 24919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:26.525698 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.525108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39444)
0620 07:06:26.525378 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.525498 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 07:06:26.526914 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:26.526449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39237 (local address 127.0.0.1:39456)
0620 07:06:26.526684 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:26.526778 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 07:06:26.568589 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24793
I20260620 07:06:26.581552 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 24415
I20260620 07:06:26.582697 24478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:26.698293 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 24415
I20260620 07:06:26.719645 11591 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:06:27.055713 11591 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:06:27.055976 11591 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:06:27.392032 11591 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:06:27.392 UTC [23468] LOG: received smart shutdown request
2026-06-20 07:06:27.395 UTC [23468] LOG: background worker "logical replication launcher" (PID 23477) exited with exit code 1
2026-06-20 07:06:27.395 UTC [23472] LOG: shutting down
2026-06-20 07:06:27.560 UTC [23468] LOG: database system is shut down
2026-06-20T07:06:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51187 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:06:27 dist-test-slave-wvz3 krb5kdc[24992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:06:27 dist-test-slave-wvz3 krb5kdc[24992](info): set up 2 sockets
Jun 20 07:06:27 dist-test-slave-wvz3 krb5kdc[24992](info): commencing operation
krb5kdc: starting...
W20260620 07:06:28.692129 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:06:28 dist-test-slave-wvz3 krb5kdc[24992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:06:28.918614 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:42123
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:29.247453 25005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:29.247778 25005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:29.247879 25005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:29.255343 25005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:29.255468 25005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:29.255515 25005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:29.255570 25005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:29.255616 25005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:29.266361 25005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:29.267807 25005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:29.269213 25005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:29.281977 25010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:29.281890 25011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:29.281919 25013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:29.283548 25005 server_base.cc:1034] running on GCE node
I20260620 07:06:29.284492 25005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:29.286161 25005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:29.287416 25005 hybrid_clock.cc:648] HybridClock initialized: now 1781939189287349 us; error 53 us; skew 500 ppm
I20260620 07:06:29.287792 25005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:29 dist-test-slave-wvz3 krb5kdc[24992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:29.295833 25005 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:29.297576 25005 webserver.cc:460] Webserver started at http://127.11.81.254:35259/ using document root <none> and password file <none>
I20260620 07:06:29.298324 25005 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:29.298456 25005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:29.298764 25005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:29.302052 25005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "ccfcd878cce643caa5a1e967241a18a1"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "3ba5794bffd1f2f4639af40724ff2b36"
server_key_iv: "729448d3736ee7f35164fc69a6292431"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.302940 25005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "ccfcd878cce643caa5a1e967241a18a1"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "3ba5794bffd1f2f4639af40724ff2b36"
server_key_iv: "729448d3736ee7f35164fc69a6292431"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.308720 25005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:06:29.312732 25020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:29.314301 25005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:06:29.314493 25005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "ccfcd878cce643caa5a1e967241a18a1"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "3ba5794bffd1f2f4639af40724ff2b36"
server_key_iv: "729448d3736ee7f35164fc69a6292431"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.314702 25005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:29.338111 25005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:29.342504 25005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:29.342782 25005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:29.360632 25005 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:42123
I20260620 07:06:29.360683 25071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:42123 every 8 connection(s)
I20260620 07:06:29.361999 25005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:29.365871 25072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:29.370522 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25005
I20260620 07:06:29.370775 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:06:29.371590 11591 external_mini_cluster.cc:1428] Setting key 118f5361d5fbd8de49b0de2d0ed5011c
I20260620 07:06:29.374101 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:06:29 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.253@KRBTEST.COM" created.
I20260620 07:06:29.398888 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.378143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:42123 (local address 127.0.0.1:40216)
0620 07:06:29.379041 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.379090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:29.381233 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:29.382099 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:29.382106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:29.382868 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:29.383364 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.383389 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.384908 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.384913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.386549 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.386568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.387593 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.387600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.387765 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.388422 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:29.388455 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:29.391473 (+ 3018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:29.394708 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.394719 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.394729 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.395134 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.395684 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.395688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.395690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.395858 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.396830 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:29.396836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:29.397059 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 07:06:29.397283 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 07:06:29.397737 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":399,"threads_started":1}
I20260620 07:06:29.398911 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.378144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.11.81.254:42123)
0620 07:06:29.379892 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:29.379901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:29.379956 (+ 55us) server_negotiation.cc:407] Connection header received
0620 07:06:29.381627 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:29.381634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:29.381871 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:29.382091 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:29.383566 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.384747 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.386696 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.387348 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.387489 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.391728 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:29.391737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:29.391750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:29.391787 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:29.394559 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.395270 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.395277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.395280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.395561 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.395958 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.395961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.395963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.396668 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:29.396872 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:29.397482 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 07:06:29.397919 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":1535,"spinlock_wait_cycles":33152,"thread_start_us":1320,"threads_started":1}
I20260620 07:06:29.402123 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:06:29.403355 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.402831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48892)
0620 07:06:29.403057 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.403181 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:06:29.403658 25022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
I20260620 07:06:29.405404 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.404975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48902)
0620 07:06:29.405178 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.405277 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:06:29.405584 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42987: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
I20260620 07:06:29.424912 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:42987
--webserver_interface=127.11.81.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:29.454939 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } attempt: 1
I20260620 07:06:29.456211 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.455739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48910)
0620 07:06:29.455950 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.456069 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:06:29.457224 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.456873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48912)
0620 07:06:29.457019 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.457094 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:06:29.457408 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42987: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
I20260620 07:06:29.525653 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } attempt: 2
I20260620 07:06:29.526814 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.526346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48928)
0620 07:06:29.526585 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.526674 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:06:29.527894 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.527520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48932)
0620 07:06:29.527677 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.527755 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 07:06:29.528093 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42987: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
I20260620 07:06:29.619369 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } attempt: 3
I20260620 07:06:29.620465 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.620008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48948)
0620 07:06:29.620241 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.620330 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 07:06:29.621551 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.621145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48962)
0620 07:06:29.621291 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.621372 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:06:29.621711 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42987: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
W20260620 07:06:29.749363 25079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:29.749689 25079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:29.749825 25079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:29.757333 25079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:29.757483 25079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:29.757568 25079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:29.757645 25079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:29.757704 25079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:29.764942 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } attempt: 4
I20260620 07:06:29.766077 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.765570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48972)
0620 07:06:29.765839 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.765937 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 07:06:29.767143 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.766785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48974)
0620 07:06:29.766929 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.767013 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 07:06:29.767349 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42987: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42987: connect: Connection refused (error 111)
I20260620 07:06:29.768237 25079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.81.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:29.769580 25079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:29.770916 25079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:29.782823 25084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:29.782997 25087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:29.782856 25085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:29.783608 25079 server_base.cc:1034] running on GCE node
I20260620 07:06:29.784364 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:29.785876 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:29.787084 25079 hybrid_clock.cc:648] HybridClock initialized: now 1781939189787027 us; error 43 us; skew 500 ppm
I20260620 07:06:29.787464 25079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:29 dist-test-slave-wvz3 krb5kdc[24992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:29.790584 25079 init.cc:377] Logged in from keytab as kudu/127.11.81.253@KRBTEST.COM (short username kudu)
I20260620 07:06:29.792212 25079 webserver.cc:460] Webserver started at http://127.11.81.253:45319/ using document root <none> and password file <none>
I20260620 07:06:29.792927 25079 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:29.793043 25079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:29.793334 25079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:29.796466 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data/instance:
uuid: "9f97682f0cab46fe800699b896346929"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "a3ed9eb22f49b0cf52c31f004e40cf7d"
server_key_iv: "c5e71c27657e66310f7f71fcc49dfddb"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.797370 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal/instance:
uuid: "9f97682f0cab46fe800699b896346929"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "a3ed9eb22f49b0cf52c31f004e40cf7d"
server_key_iv: "c5e71c27657e66310f7f71fcc49dfddb"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.802726 25079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:06:29.806541 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:29.807998 25079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:06:29.808161 25079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "9f97682f0cab46fe800699b896346929"
format_stamp: "Formatted at 2026-06-20 07:06:29 on dist-test-slave-wvz3"
server_key: "a3ed9eb22f49b0cf52c31f004e40cf7d"
server_key_iv: "c5e71c27657e66310f7f71fcc49dfddb"
server_key_version: "encryptionkey@0"
I20260620 07:06:29.808346 25079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:29.827406 25079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:29.831794 25079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:29.832057 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:29.849982 25079 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.253:42987
I20260620 07:06:29.850029 25145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.253:42987 every 8 connection(s)
I20260620 07:06:29.851369 25079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
I20260620 07:06:29.854948 25146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:29.856009 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25079
I20260620 07:06:29.856207 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-1/wal/instance
I20260620 07:06:29.856630 11591 external_mini_cluster.cc:1428] Setting key 89c7b49805639ae578e9352a646ae557
I20260620 07:06:29.861675 25146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.81.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.252@KRBTEST.COM" created.
Jun 20 07:06:29 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:29.885042 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.864842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.11.81.254:42123)
0620 07:06:29.865056 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:29.865063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:29.866381 (+ 1318us) server_negotiation.cc:407] Connection header received
0620 07:06:29.868358 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:29.868364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:29.868459 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:29.868578 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:29.870075 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.871071 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.872720 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.873354 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.873508 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.879074 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:29.879080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:29.879083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:29.879123 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:29.881800 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.882664 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.882668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.882670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.882796 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.883229 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.883233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.883235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.883571 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:29.883714 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:29.884484 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 07:06:29.884729 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:06:29.886066 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.865378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:42123 (local address 127.0.0.1:40226)
0620 07:06:29.866195 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.866248 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:29.868085 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:29.868781 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:29.868790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:29.869309 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:29.869896 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.869915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.871244 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.871249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.872561 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.872575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.873598 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.873603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.873801 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.874445 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:29.874469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:29.878785 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:29.882019 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.882027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.882042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.882523 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.882910 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.882915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.882918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.883109 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.883706 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:29.883715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:29.884098 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 07:06:29.884339 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 07:06:29.884894 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260620 07:06:29.887964 25146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:06:29 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:06:29.902247 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.888637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48982)
0620 07:06:29.888807 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.888830 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:29.888941 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:29.890080 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:29.890084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:29.890453 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:29.890595 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.890602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.892430 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.892434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.893261 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:29.893272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.894092 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.894098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:29.894304 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.894893 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:29.894910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:29.897062 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:29.900094 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.900098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.900100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.900450 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.900779 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:29.900783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:29.900785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:29.900956 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:29.901507 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:29.901511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:29.901641 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 07:06:29.901752 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 07:06:29.901990 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:06:29.902491 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.888850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.11.81.253:42987)
0620 07:06:29.889586 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:29.889593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:29.889624 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:06:29.889675 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:29.889683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:29.889922 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:29.890070 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:29.890758 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.892290 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.893391 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:29.893940 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:29.894075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:29.897257 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:29.897268 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:29.897282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:29.897315 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:29.899972 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.900564 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.900571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.900573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.900688 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:29.901045 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:29.901048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:29.901050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:29.901407 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:29.901574 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:29.901886 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 07:06:29.902135 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":412,"threads_started":1}
I20260620 07:06:29.904832 25146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
I20260620 07:06:29.906143 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.905626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59880)
0620 07:06:29.905812 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.905952 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 07:06:29.906405 25098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:29.906781 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:41185
--webserver_interface=127.11.81.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:29.907995 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.907661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59892)
0620 07:06:29.907809 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.907888 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:06:29.908226 25146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:29.957545 25146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 1
I20260620 07:06:29.958858 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.958316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59894)
0620 07:06:29.958599 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.958706 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 07:06:29.959880 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:29.959546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59900)
0620 07:06:29.959700 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:29.959771 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:06:29.960083 25146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.028298 25146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 2
I20260620 07:06:30.029435 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.028973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59916)
0620 07:06:30.029171 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.029270 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:06:30.030514 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.030183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59920)
0620 07:06:30.030316 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.030394 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:06:30.030759 25146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.066627 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } attempt: 5
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:06:30.081161 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.067536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.11.81.253:42987)
0620 07:06:30.067708 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.067713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.067732 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:06:30.067918 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.067922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.068055 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.068138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.069193 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.069866 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.071532 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.072008 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.072103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.075831 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.075836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.075840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.075875 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.078571 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.079252 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.079257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.079259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.079421 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.079792 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.079797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.079801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.080097 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.080230 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.080639 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.080850 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:06:30.081163 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.067354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48984)
0620 07:06:30.067609 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.067636 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.067788 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.068222 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.068227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.068768 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.069011 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.069024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.070007 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.070013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.071384 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.071400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.072155 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.072161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.072345 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.073083 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.073107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.075625 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.078732 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.078738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.078743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.079121 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.079528 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.079533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.079545 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.079680 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.080213 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.080218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.080365 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.080503 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.080803 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:06:30.082095 25072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:06:30.083189 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.082792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59934)
0620 07:06:30.082959 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.083046 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:06:30.084192 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.083859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59936)
0620 07:06:30.084000 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.084073 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:06:30.084375 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.121986 25146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 3
I20260620 07:06:30.123162 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.122678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59946)
0620 07:06:30.122896 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.123005 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:06:30.124260 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.123810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59950)
0620 07:06:30.124055 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.124137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260620 07:06:30.124505 25146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.135584 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 1
I20260620 07:06:30.136706 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.136249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59954)
0620 07:06:30.136445 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.136539 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:06:30.137761 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.137391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59956)
0620 07:06:30.137563 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.137649 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:06:30.137948 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.206211 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 2
I20260620 07:06:30.207572 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.206950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59968)
0620 07:06:30.207247 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.207394 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 07:06:30.208647 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.208283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59976)
0620 07:06:30.208437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.208512 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:06:30.208858 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
W20260620 07:06:30.238073 25153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:30.238389 25153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:30.238533 25153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:30.245927 25153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:30.246070 25153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:30.246152 25153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:30.246227 25153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:30.246294 25153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:30.256651 25153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:42123,127.11.81.253:42987,127.11.81.252:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.81.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:30.258025 25153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:30.259384 25153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:06:30.267694 25146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 4
I20260620 07:06:30.268805 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.268319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59982)
0620 07:06:30.268561 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.268650 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:06:30.269872 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.269438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59988)
0620 07:06:30.269654 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.269739 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 07:06:30.270238 25146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
W20260620 07:06:30.270747 25158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:30.270780 25161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:30.270828 25159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:30.272269 25153 server_base.cc:1034] running on GCE node
I20260620 07:06:30.273032 25153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:30.274446 25153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:30.275632 25153 hybrid_clock.cc:648] HybridClock initialized: now 1781939190275573 us; error 47 us; skew 500 ppm
I20260620 07:06:30.275939 25153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939190, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:30.278908 25153 init.cc:377] Logged in from keytab as kudu/127.11.81.252@KRBTEST.COM (short username kudu)
I20260620 07:06:30.280376 25153 webserver.cc:460] Webserver started at http://127.11.81.252:44921/ using document root <none> and password file <none>
I20260620 07:06:30.281045 25153 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:30.281147 25153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:30.281443 25153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:30.284410 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data/instance:
uuid: "673f3110f2464ebbb8617e68141162f5"
format_stamp: "Formatted at 2026-06-20 07:06:30 on dist-test-slave-wvz3"
server_key: "e27e3f69cd270e95e1eb75e6812eb177"
server_key_iv: "bc29cac52943b4063d231ab8b3239593"
server_key_version: "encryptionkey@0"
I20260620 07:06:30.285213 25153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal/instance:
uuid: "673f3110f2464ebbb8617e68141162f5"
format_stamp: "Formatted at 2026-06-20 07:06:30 on dist-test-slave-wvz3"
server_key: "e27e3f69cd270e95e1eb75e6812eb177"
server_key_iv: "bc29cac52943b4063d231ab8b3239593"
server_key_version: "encryptionkey@0"
I20260620 07:06:30.290232 25153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:06:30.293651 25168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:30.295010 25153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:06:30.295181 25153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "673f3110f2464ebbb8617e68141162f5"
format_stamp: "Formatted at 2026-06-20 07:06:30 on dist-test-slave-wvz3"
server_key: "e27e3f69cd270e95e1eb75e6812eb177"
server_key_iv: "bc29cac52943b4063d231ab8b3239593"
server_key_version: "encryptionkey@0"
I20260620 07:06:30.295415 25153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:30.315099 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 3
I20260620 07:06:30.316262 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.315779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:59990)
0620 07:06:30.315984 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.316078 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 07:06:30.317258 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.316870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:60002)
0620 07:06:30.317073 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.317155 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 07:06:30.317459 25072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:41185: Network error: Client connection negotiation failed: client connection to 127.11.81.252:41185: connect: Connection refused (error 111)
I20260620 07:06:30.324790 25153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:30.329145 25153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:30.329414 25153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:30.348067 25153 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.252:41185
I20260620 07:06:30.348145 25219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.252:41185 every 8 connection(s)
I20260620 07:06:30.349454 25153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
I20260620 07:06:30.353272 25220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:30.358628 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25153
I20260620 07:06:30.358798 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938992575046-11591-0/minicluster-data/master-2/wal/instance
I20260620 07:06:30.359519 11591 external_mini_cluster.cc:1428] Setting key c8541543e70d24bfcbc15fccab049b5d
I20260620 07:06:30.359787 25220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939190, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
I20260620 07:06:30.382064 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.363787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.11.81.254:42123)
0620 07:06:30.363976 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.363981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.364734 (+ 753us) server_negotiation.cc:407] Connection header received
0620 07:06:30.365868 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.365875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.365957 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.366066 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.367876 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.368982 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.370839 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.371502 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.371614 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.375505 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.375511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.375515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.375551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.378756 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.379670 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.379675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.379677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.379813 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.380244 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.380247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.380250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.380572 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.380726 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.381454 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.381699 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:06:30.383095 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.363635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:42123 (local address 127.0.0.1:40236)
0620 07:06:30.364581 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.364637 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.365562 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.366303 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.366309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.367108 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.367676 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.367705 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.369159 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.369164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.370708 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.370720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.371731 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.371739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.371903 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.372483 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.372513 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.375234 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.378899 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.378909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.378918 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.379560 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.379936 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.379946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.379948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.380133 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.380692 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.380699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.381067 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.381340 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.381877 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":39,"thread_start_us":516,"threads_started":1}
I20260620 07:06:30.383131 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.361677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.11.81.252:41185)
0620 07:06:30.363275 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.363312 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.363358 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:06:30.365547 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.365575 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.365883 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.366067 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.367593 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.369256 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.370571 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.371279 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.371491 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.375943 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.375955 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.375972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.376007 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.379060 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.379870 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.379880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.379883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.380049 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.380423 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.380427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.380430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.381062 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.381278 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.381693 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.382005 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":1373,"thread_start_us":1135,"threads_started":1}
I20260620 07:06:30.385524 25220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939190, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:30.400388 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.386145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42987 (local address 127.0.0.1:48990)
0620 07:06:30.386366 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.386383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.386484 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.386910 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.386918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.387444 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.387670 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.387684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.388772 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.388776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.389967 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.389984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.390738 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.390744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.390952 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.391567 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.391590 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.394091 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.397751 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.397755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.397758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.398185 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.398671 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.398675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.398677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.398830 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.399450 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.399455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.399671 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.399830 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.400106 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 07:06:30.400566 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.386360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.11.81.253:42987)
0620 07:06:30.386510 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.386515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.386532 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:06:30.386587 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.386593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.386671 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.386776 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.387826 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.388620 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.390134 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.390579 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.390694 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.394313 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.394320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.394324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.394386 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.397584 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.398318 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.398324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.398328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.398540 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.398937 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.398942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.398945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.399306 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.399449 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.399942 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.400172 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:06:30.401540 25220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:06:30.401888 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.386239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.11.81.254:42123)
0620 07:06:30.386396 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.386402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.386503 (+ 101us) server_negotiation.cc:407] Connection header received
0620 07:06:30.386642 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.386647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.386708 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.386790 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.388463 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.389262 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.390931 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.391442 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.391547 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.395849 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.395856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.395860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.395904 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.398940 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.399787 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.399792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.399794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.399979 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.400487 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.400493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.400497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.400860 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.401131 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.401433 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.401623 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939190, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:06:30.415210 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.402490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:60030)
0620 07:06:30.402609 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.402627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.402729 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.403068 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.403073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.403543 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.403722 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.403734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.404768 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.404773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.405836 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.405848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.406718 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.406724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.406916 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.407512 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.407535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.409678 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.412761 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.412766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.412771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.413170 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.413540 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.413545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.413547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.413672 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.414222 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.414226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.414370 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.414495 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.414779 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 07:06:30.415244 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.402432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.11.81.252:41185)
0620 07:06:30.402599 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.402604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.402673 (+ 69us) server_negotiation.cc:407] Connection header received
0620 07:06:30.402835 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.402840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.402913 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.403013 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.403847 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.404636 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.405989 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.406584 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.406668 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.409864 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.409868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.409872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.409902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.412609 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.413274 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.413279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.413283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.413419 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.413774 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.413779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.413782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.414113 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.414245 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.414593 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.414944 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":50}
I20260620 07:06:30.418952 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.403613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.11.81.253:42987)
0620 07:06:30.403767 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.403771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.403787 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:06:30.403935 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.403941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.404008 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.404094 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.405380 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.406315 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.407964 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.408453 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.408574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.413556 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.413564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.413569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.413606 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.416360 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.417062 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.417069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.417072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.417211 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.417557 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.417561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.417563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.417845 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.418103 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.418407 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.418644 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:06:30.420280 25220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: Bootstrap starting.
I20260620 07:06:30.423847 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:06:30.424178 25220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:30.425436 25220 log.cc:824] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:30.428923 25220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: No bootstrap required, opened a new log
I20260620 07:06:30.440521 25220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.441165 25220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:30.441340 25220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673f3110f2464ebbb8617e68141162f5, State: Initialized, Role: FOLLOWER
I20260620 07:06:30.442250 25220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.448582 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.429458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.11.81.254:42123)
0620 07:06:30.429640 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.429646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.430325 (+ 679us) server_negotiation.cc:407] Connection header received
0620 07:06:30.430764 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.430772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.430866 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.430950 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.433588 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.434706 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.437035 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.437671 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.437768 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.441736 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.441744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.441749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.441800 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.445214 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.446145 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.446152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.446156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.446320 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.446793 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.446798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.446802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.447156 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.447429 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.447974 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.448235 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:06:30.449577 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.450004 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.452049 25220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:30.457510 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.429415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.11.81.252:41185)
0620 07:06:30.429775 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.429781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.439850 (+ 10069us) server_negotiation.cc:407] Connection header received
0620 07:06:30.440015 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.440021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.440123 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.440231 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.443038 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.444245 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.446408 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.447148 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.447244 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.450871 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.450878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.450882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.450924 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.454547 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.455259 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.455265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.455269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.455456 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.455884 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.455889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.455892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.456318 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.456576 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.456856 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.457087 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260620 07:06:30.458076 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.430897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.11.81.253:42987)
0620 07:06:30.431088 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.431095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.442264 (+ 11169us) server_negotiation.cc:407] Connection header received
0620 07:06:30.442386 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.442393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.442472 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.442582 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.444298 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.445367 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.447359 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.447952 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.448039 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.451096 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.451102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.451107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.451149 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.454348 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.455482 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.455488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.455492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.455640 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.456037 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.456042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.456045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.456330 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.456568 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.457578 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.457808 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:06:30.467623 25244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:06:30.467756 25244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:06:30.465384 25153 master_runner.cc:186] Error getting master registration for 127.11.81.254:42123: OK, instance_id { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" instance_seqno: 1781939189355313 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:06:30.477077 25153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:06:30.494660 25072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 4
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
I20260620 07:06:30.509985 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.495311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:60050)
0620 07:06:30.495600 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.495619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.495740 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.496277 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.496282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.496747 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.496968 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.496981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.498421 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.498426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.499433 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.499446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.500410 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.500414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.500558 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.501168 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.501184 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.503408 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.507184 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.507188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.507191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.507710 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.508199 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.508203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.508207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.508377 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.509126 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.509131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.509306 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.509449 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.509723 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 07:06:30.510205 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.495368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.11.81.252:41185)
0620 07:06:30.495576 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.495582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.495845 (+ 263us) server_negotiation.cc:407] Connection header received
0620 07:06:30.495899 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.495903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.496006 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.496099 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.497119 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.498278 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.499563 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.500284 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.500409 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.503594 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.503601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.503607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.503647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.507043 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.507859 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.507864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.507868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.508062 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.508507 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.508512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.508515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.508923 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.509060 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.509585 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.509830 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:06:30.516024 25072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: Bootstrap starting.
I20260620 07:06:30.520269 25072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:30.521733 25072 log.cc:824] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:30.524988 25072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: No bootstrap required, opened a new log
I20260620 07:06:30.535918 25072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.536270 25072 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:30.536387 25072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfcd878cce643caa5a1e967241a18a1, State: Initialized, Role: FOLLOWER
I20260620 07:06:30.536937 25072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.537854 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.538254 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.538861 25072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:06:30.551599 25258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:06:30.551827 25258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:06:30.549041 25005 master_runner.cc:186] Error getting master registration for 127.11.81.253:42987: OK, instance_id { permanent_uuid: "9f97682f0cab46fe800699b896346929" instance_seqno: 1781939189845027 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:06:30.560407 25005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:06:30.569475 25146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } attempt: 5
Jun 20 07:06:30 dist-test-slave-wvz3 krb5kdc[24992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939189, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
W20260620 07:06:30.581555 25174 tablet.cc:2369] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:06:30.589090 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.570304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:41185 (local address 127.0.0.1:60054)
0620 07:06:30.570634 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:30.570656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:30.570786 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:30.571390 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:30.571396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:30.571920 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:30.572152 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.572166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.573613 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.573619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.575034 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:30.575051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.576076 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.576083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:30.576328 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.577013 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:30.577039 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:30.581888 (+ 4849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:30.586118 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.586124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.586128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.586679 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.587117 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:30.587123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:30.587127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:30.587322 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:30.588003 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:30.588009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:30.588190 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 07:06:30.588350 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:06:30.588670 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 07:06:30.589134 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:30.570395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.11.81.252:41185)
0620 07:06:30.571029 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:30.571047 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:30.571067 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:06:30.571132 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:30.571137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:30.571214 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:30.571350 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:30.572327 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.573458 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.575199 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:30.575912 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:30.576056 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:30.582154 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:30.582162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:30.582167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:30.582228 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:30.585932 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.586817 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.586823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.586827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.586982 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:30.587438 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:30.587444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:30.587448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:30.587825 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:30.587956 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:30.588488 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 07:06:30.588746 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492}
W20260620 07:06:30.595721 25026 tablet.cc:2369] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:06:30.595821 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: Bootstrap starting.
I20260620 07:06:30.601779 25146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:30.603166 25146 log.cc:824] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:30.606693 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: No bootstrap required, opened a new log
I20260620 07:06:30.617990 25146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.618615 25146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:30.618777 25146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f97682f0cab46fe800699b896346929, State: Initialized, Role: FOLLOWER
I20260620 07:06:30.619593 25146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.621176 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.621608 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.622498 25146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:30.639008 25079 master_runner.cc:263] No leader master found from master 9f97682f0cab46fe800699b896346929
I20260620 07:06:30.639175 25079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:06:30.642201 25272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:06:30.642445 25272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:06:30.689847 25233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:06:30.690280 25233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.692305 25233 leader_election.cc:288] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccfcd878cce643caa5a1e967241a18a1 (127.11.81.254:42123), 9f97682f0cab46fe800699b896346929 (127.11.81.253:42987)
I20260620 07:06:30.693585 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673f3110f2464ebbb8617e68141162f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f97682f0cab46fe800699b896346929" is_pre_election: true
I20260620 07:06:30.694260 25121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673f3110f2464ebbb8617e68141162f5 in term 0.
I20260620 07:06:30.694682 25261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:06:30.694887 25261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.695884 25170 leader_election.cc:302] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 673f3110f2464ebbb8617e68141162f5, 9f97682f0cab46fe800699b896346929; no voters:
I20260620 07:06:30.697335 25261 leader_election.cc:288] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccfcd878cce643caa5a1e967241a18a1 (127.11.81.254:42123), 673f3110f2464ebbb8617e68141162f5 (127.11.81.252:41185)
I20260620 07:06:30.698421 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f97682f0cab46fe800699b896346929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfcd878cce643caa5a1e967241a18a1" is_pre_election: true
I20260620 07:06:30.699018 25047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f97682f0cab46fe800699b896346929 in term 0.
I20260620 07:06:30.699819 25096 leader_election.cc:302] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f97682f0cab46fe800699b896346929, ccfcd878cce643caa5a1e967241a18a1; no voters:
I20260620 07:06:30.700937 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f97682f0cab46fe800699b896346929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673f3110f2464ebbb8617e68141162f5" is_pre_election: true
I20260620 07:06:30.701416 25195 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f97682f0cab46fe800699b896346929 in term 0.
I20260620 07:06:30.702365 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673f3110f2464ebbb8617e68141162f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfcd878cce643caa5a1e967241a18a1" is_pre_election: true
I20260620 07:06:30.702612 25047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673f3110f2464ebbb8617e68141162f5 in term 0.
I20260620 07:06:30.707206 25261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:06:30.707664 25261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:06:30.707775 25261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:30.711043 25261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.711575 25233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:06:30.711761 25233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:06:30.711862 25233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:30.716573 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f97682f0cab46fe800699b896346929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfcd878cce643caa5a1e967241a18a1"
I20260620 07:06:30.716812 25047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:30.720693 25047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f97682f0cab46fe800699b896346929 in term 1.
I20260620 07:06:30.721324 25096 leader_election.cc:302] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f97682f0cab46fe800699b896346929, ccfcd878cce643caa5a1e967241a18a1; no voters:
I20260620 07:06:30.724741 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f97682f0cab46fe800699b896346929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673f3110f2464ebbb8617e68141162f5"
I20260620 07:06:30.715157 25233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.725708 25261 leader_election.cc:288] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [CANDIDATE]: Term 1 election: Requested vote from peers ccfcd878cce643caa5a1e967241a18a1 (127.11.81.254:42123), 673f3110f2464ebbb8617e68141162f5 (127.11.81.252:41185)
I20260620 07:06:30.726493 25195 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f97682f0cab46fe800699b896346929 in current term 1: Already voted for candidate 673f3110f2464ebbb8617e68141162f5 in this term.
I20260620 07:06:30.733300 25047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673f3110f2464ebbb8617e68141162f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfcd878cce643caa5a1e967241a18a1"
I20260620 07:06:30.733762 25047 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 673f3110f2464ebbb8617e68141162f5 in current term 1: Already voted for candidate 9f97682f0cab46fe800699b896346929 in this term.
I20260620 07:06:30.733934 25261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:30.734937 25261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [term 1 LEADER]: Becoming Leader. State: Replica: 9f97682f0cab46fe800699b896346929, State: Running, Role: LEADER
I20260620 07:06:30.736008 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673f3110f2464ebbb8617e68141162f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f97682f0cab46fe800699b896346929"
I20260620 07:06:30.736999 25261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } }
I20260620 07:06:30.737480 25233 leader_election.cc:288] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [CANDIDATE]: Term 1 election: Requested vote from peers ccfcd878cce643caa5a1e967241a18a1 (127.11.81.254:42123), 9f97682f0cab46fe800699b896346929 (127.11.81.253:42987)
I20260620 07:06:30.758438 25170 leader_election.cc:302] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673f3110f2464ebbb8617e68141162f5; no voters: 9f97682f0cab46fe800699b896346929, ccfcd878cce643caa5a1e967241a18a1
I20260620 07:06:30.759047 25233 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 07:06:30.759857 25274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97682f0cab46fe800699b896346929. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.760362 25274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:30.766759 25276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:30.769194 25276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:30.793627 25195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [term 1 FOLLOWER]: Refusing update from remote peer 9f97682f0cab46fe800699b896346929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:30.794718 25274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:30.802532 25047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [term 1 FOLLOWER]: Refusing update from remote peer 9f97682f0cab46fe800699b896346929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:30.803620 25274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:30.815543 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97682f0cab46fe800699b896346929. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.815845 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.820639 25279 mvcc.cc:205] Tried to move back new op lower bound from 7298822925476999168 to 7298822925274857472. Current Snapshot: MvccSnapshot[applied={T|T < 7298822925476999168}]
I20260620 07:06:30.827083 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f97682f0cab46fe800699b896346929. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.827443 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.827775 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.828022 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.829640 25274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.829927 25274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:30.830276 25274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.830493 25274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929 [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:30.833029 25276 catalog_manager.cc:1347] Generated new cluster ID: 81a8ec6cd5bd4fe6889bee92c0bfe8de
I20260620 07:06:30.833134 25276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:30.839702 25233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f97682f0cab46fe800699b896346929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfcd878cce643caa5a1e967241a18a1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42123 } } peers { permanent_uuid: "9f97682f0cab46fe800699b896346929" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42987 } } peers { permanent_uuid: "673f3110f2464ebbb8617e68141162f5" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 41185 } } }
I20260620 07:06:30.840025 25233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:06:30.872372 25276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:06:30.874326 25276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:30.895738 25276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f97682f0cab46fe800699b896346929: Generated new TSK 0
I20260620 07:06:30.896973 25276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:06:31.469725 25244 catalog_manager.cc:1259] Loaded cluster ID: 81a8ec6cd5bd4fe6889bee92c0bfe8de
I20260620 07:06:31.469823 25244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: loading cluster ID for follower catalog manager: success
I20260620 07:06:31.472311 25244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: acquiring CA information for follower catalog manager: success
I20260620 07:06:31.474073 25244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 673f3110f2464ebbb8617e68141162f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:06:31.553869 25258 catalog_manager.cc:1259] Loaded cluster ID: 81a8ec6cd5bd4fe6889bee92c0bfe8de
I20260620 07:06:31.553972 25258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: loading cluster ID for follower catalog manager: success
I20260620 07:06:31.556411 25258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: acquiring CA information for follower catalog manager: success
I20260620 07:06:31.558152 25258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccfcd878cce643caa5a1e967241a18a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:06:32.012981 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25005
I20260620 07:06:32.014281 25067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:32.125334 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25005
I20260620 07:06:32.142481 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25079
I20260620 07:06:32.143649 25141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:32.251966 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25079
I20260620 07:06:32.269528 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25153
I20260620 07:06:32.270725 25215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:32.377322 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25153
2026-06-20T07:06:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4709 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:06:32 dist-test-slave-wvz3 krb5kdc[25297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:06:32 dist-test-slave-wvz3 krb5kdc[25297](info): set up 2 sockets
Jun 20 07:06:32 dist-test-slave-wvz3 krb5kdc[25297](info): commencing operation
krb5kdc: starting...
W20260620 07:06:33.387634 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:06:33 dist-test-slave-wvz3 krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:06:33.615315 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:34489
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:33.944878 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:33.945201 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:33.945338 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:33.952826 25310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:33.952958 25310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:33.953013 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:33.953094 25310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:33.953150 25310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:33.963485 25310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:33.964910 25310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:33.966355 25310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:33.978166 25315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:33.978255 25316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:33.978286 25318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:33.979673 25310 server_base.cc:1034] running on GCE node
I20260620 07:06:33.980473 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:33.981881 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:33.983110 25310 hybrid_clock.cc:648] HybridClock initialized: now 1781939193983056 us; error 46 us; skew 500 ppm
I20260620 07:06:33.983541 25310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:33 dist-test-slave-wvz3 krb5kdc[25297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939193, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:33.986773 25310 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:33.988479 25310 webserver.cc:460] Webserver started at http://127.11.81.254:33773/ using document root <none> and password file <none>
I20260620 07:06:33.989182 25310 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:33.989300 25310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:33.989604 25310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:33.992700 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "df1e27b5cb134215b61d850c06aa5850"
format_stamp: "Formatted at 2026-06-20 07:06:33 on dist-test-slave-wvz3"
server_key: "631be8c45f6cfd20eee0e48c610c3cb1"
server_key_iv: "42183b863cf960f4ceb4ce93a68417b8"
server_key_version: "encryptionkey@0"
I20260620 07:06:33.993487 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "df1e27b5cb134215b61d850c06aa5850"
format_stamp: "Formatted at 2026-06-20 07:06:33 on dist-test-slave-wvz3"
server_key: "631be8c45f6cfd20eee0e48c610c3cb1"
server_key_iv: "42183b863cf960f4ceb4ce93a68417b8"
server_key_version: "encryptionkey@0"
I20260620 07:06:33.999004 25310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:06:34.002728 25325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:34.004195 25310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:06:34.004369 25310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "df1e27b5cb134215b61d850c06aa5850"
format_stamp: "Formatted at 2026-06-20 07:06:33 on dist-test-slave-wvz3"
server_key: "631be8c45f6cfd20eee0e48c610c3cb1"
server_key_iv: "42183b863cf960f4ceb4ce93a68417b8"
server_key_version: "encryptionkey@0"
I20260620 07:06:34.004563 25310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:34.018404 25310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:34.022734 25310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:34.023053 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:34.040138 25310 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:34489
I20260620 07:06:34.040141 25376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:34489 every 8 connection(s)
I20260620 07:06:34.041527 25310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:34.045202 25377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:34.047267 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25310
I20260620 07:06:34.047515 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:06:34.048316 11591 external_mini_cluster.cc:1428] Setting key 4931c2ee7546d70ac4cacea64b26169b
I20260620 07:06:34.058461 25377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850: Bootstrap starting.
I20260620 07:06:34.062281 25377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:34.063442 25377 log.cc:824] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:06:34 dist-test-slave-wvz3 krb5kdc[25297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:34.066738 25377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850: No bootstrap required, opened a new log
I20260620 07:06:34.073662 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:34.050486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.11.81.254:34489)
0620 07:06:34.051439 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:34.051457 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:34.051821 (+ 364us) server_negotiation.cc:407] Connection header received
0620 07:06:34.054193 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:34.054224 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:34.054785 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:34.055264 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:34.056717 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:34.058676 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:34.060140 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:34.061281 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:34.061513 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:34.066034 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:34.066042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:34.066057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:34.066097 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:34.069281 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:34.070176 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:34.070182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:34.070189 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:34.070359 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:34.070791 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:34.070793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:34.070795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:34.071380 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:34.071618 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:34.072116 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:06:34.072508 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":365,"threads_started":1}
I20260620 07:06:34.075742 25377 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } }
I20260620 07:06:34.076215 25377 raft_consensus.cc:374] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:34.076362 25377 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1e27b5cb134215b61d850c06aa5850, State: Initialized, Role: FOLLOWER
I20260620 07:06:34.077020 25377 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } }
I20260620 07:06:34.077287 25377 raft_consensus.cc:388] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:34.077422 25377 raft_consensus.cc:482] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:34.077610 25377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:34.078276 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:06:34.080747 25377 raft_consensus.cc:504] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } }
I20260620 07:06:34.081357 25377 leader_election.cc:302] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df1e27b5cb134215b61d850c06aa5850; no voters:
I20260620 07:06:34.082021 25377 leader_election.cc:288] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:06:34.082115 25382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:34.082906 25382 raft_consensus.cc:686] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [term 1 LEADER]: Becoming Leader. State: Replica: df1e27b5cb134215b61d850c06aa5850, State: Running, Role: LEADER
I20260620 07:06:34.083561 25382 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } }
I20260620 07:06:34.084268 25377 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:34.086360 25384 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1e27b5cb134215b61d850c06aa5850. Latest consensus state: current_term: 1 leader_uuid: "df1e27b5cb134215b61d850c06aa5850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } } }
I20260620 07:06:34.086376 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df1e27b5cb134215b61d850c06aa5850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df1e27b5cb134215b61d850c06aa5850" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 34489 } } }
I20260620 07:06:34.086800 25384 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:34.086800 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850 [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:34.088946 25389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:34.093441 25389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:34.102095 25389 catalog_manager.cc:1347] Generated new cluster ID: 6433131d0bfd48a7b0b711185e197baf
I20260620 07:06:34.102224 25389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:34.129493 25389 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:06:34.131552 25389 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:34.145591 25389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df1e27b5cb134215b61d850c06aa5850: Generated new TSK 0
I20260620 07:06:34.146725 25389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:06:34.192023 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25310
I20260620 07:06:34.193359 25372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:34.293144 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25310
2026-06-20T07:06:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1912 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:06:34 dist-test-slave-wvz3 krb5kdc[25405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:06:34 dist-test-slave-wvz3 krb5kdc[25405](info): set up 2 sockets
Jun 20 07:06:34 dist-test-slave-wvz3 krb5kdc[25405](info): commencing operation
krb5kdc: starting...
W20260620 07:06:35.311714 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:06:35 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:06:35.540940 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:35.861321 25418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:35.861635 25418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:35.861765 25418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:35.869019 25418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:35.869149 25418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:35.869230 25418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:35.869303 25418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:35.869360 25418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:35.879513 25418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:35.880848 25418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:35.882211 25418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:35.893546 25424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:35.893447 25426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:35.893575 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:35.894173 25418 server_base.cc:1034] running on GCE node
I20260620 07:06:35.894946 25418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:35.896371 25418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:35.897586 25418 hybrid_clock.cc:648] HybridClock initialized: now 1781939195897524 us; error 50 us; skew 500 ppm
I20260620 07:06:35.897923 25418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:35 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:35.901031 25418 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:35.902537 25418 webserver.cc:460] Webserver started at http://127.11.81.254:42017/ using document root <none> and password file <none>
I20260620 07:06:35.903229 25418 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:35.903383 25418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:35.903673 25418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:35.906723 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "b80dc0794f444f4eab2f1e438287cbfb"
format_stamp: "Formatted at 2026-06-20 07:06:35 on dist-test-slave-wvz3"
server_key: "13d285dbcf709126dc020e684d7fe22c"
server_key_iv: "0f9a878349b8801ecce09ff229ea182c"
server_key_version: "encryptionkey@0"
I20260620 07:06:35.907580 25418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "b80dc0794f444f4eab2f1e438287cbfb"
format_stamp: "Formatted at 2026-06-20 07:06:35 on dist-test-slave-wvz3"
server_key: "13d285dbcf709126dc020e684d7fe22c"
server_key_iv: "0f9a878349b8801ecce09ff229ea182c"
server_key_version: "encryptionkey@0"
I20260620 07:06:35.912842 25418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:06:35.916568 25433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:35.918123 25418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:06:35.918308 25418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b80dc0794f444f4eab2f1e438287cbfb"
format_stamp: "Formatted at 2026-06-20 07:06:35 on dist-test-slave-wvz3"
server_key: "13d285dbcf709126dc020e684d7fe22c"
server_key_iv: "0f9a878349b8801ecce09ff229ea182c"
server_key_version: "encryptionkey@0"
I20260620 07:06:35.918536 25418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:35.933646 25418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:35.937908 25418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:35.938176 25418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:35.955341 25418 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43651
I20260620 07:06:35.955402 25484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43651 every 8 connection(s)
I20260620 07:06:35.956707 25418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:35.960608 25485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:35.962021 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25418
I20260620 07:06:35.962212 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:06:35.962826 11591 external_mini_cluster.cc:1428] Setting key 39f8aff1e55abb0cf62824426755c806
I20260620 07:06:35.974682 25485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap starting.
Jun 20 07:06:35 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:35.977934 25485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:35.979148 25485 log.cc:824] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:35.982883 25485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: No bootstrap required, opened a new log
I20260620 07:06:35.987172 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:35.964784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.11.81.254:43651)
0620 07:06:35.965601 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:35.965618 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:35.965672 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:06:35.967333 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:35.967361 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:35.967909 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:35.968352 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:35.969658 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:35.971714 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:35.973065 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:35.974067 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:35.974250 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:35.978469 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:35.978481 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:35.978498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:35.978543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:35.982571 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:35.983336 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:35.983344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:35.983352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:35.983567 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:35.983938 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:35.983943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:35.983946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:35.984632 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:35.984941 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:35.985340 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:06:35.985805 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":321,"threads_started":1}
I20260620 07:06:35.992151 25485 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:35.992476 25485 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:35.992559 25485 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Initialized, Role: FOLLOWER
I20260620 07:06:35.993063 25485 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:35.993256 25485 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:35.993346 25485 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:35.993467 25485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:35.995585 25485 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:35.996064 25485 leader_election.cc:302] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80dc0794f444f4eab2f1e438287cbfb; no voters:
I20260620 07:06:35.996698 25485 leader_election.cc:288] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:06:35.996798 25490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:35.997401 25490 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 LEADER]: Becoming Leader. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Running, Role: LEADER
I20260620 07:06:35.997925 25490 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:35.998893 25485 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:36.001113 25492 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80dc0794f444f4eab2f1e438287cbfb. Latest consensus state: current_term: 1 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:36.001437 25492 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:36.000698 25491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:36.001803 25491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:36.003785 25498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:36.009470 25498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:36.018281 25498 catalog_manager.cc:1347] Generated new cluster ID: 8a437bb2b703475c94b4d53c7aaaef32
I20260620 07:06:36.018410 25498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:36.059849 25498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:06:36.061800 25498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:36.076162 25498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Generated new TSK 0
I20260620 07:06:36.077265 25498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:06:36.134897 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:36.466063 25511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:36.466336 25511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:36.466454 25511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:36.473697 25511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:36.473809 25511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:36.473963 25511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:06:36.483889 25511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:36.485170 25511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:36.486691 25511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:36.499517 25517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:36.499593 25516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:36.499550 25519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:36.500433 25511 server_base.cc:1034] running on GCE node
I20260620 07:06:36.500990 25511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:36.502063 25511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:36.503254 25511 hybrid_clock.cc:648] HybridClock initialized: now 1781939196503192 us; error 61 us; skew 500 ppm
I20260620 07:06:36.503621 25511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:36 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939196, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:36.506712 25511 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:06:36.508188 25511 webserver.cc:460] Webserver started at http://127.11.81.193:33243/ using document root <none> and password file <none>
I20260620 07:06:36.508880 25511 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:36.508994 25511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:36.509279 25511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:36.512302 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
format_stamp: "Formatted at 2026-06-20 07:06:36 on dist-test-slave-wvz3"
server_key: "78e301e4b4c4009d7d98cdde0a98fed4"
server_key_iv: "83b9edcfd5dfedb4bd79bf0ef7603845"
server_key_version: "encryptionkey@0"
I20260620 07:06:36.513098 25511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
format_stamp: "Formatted at 2026-06-20 07:06:36 on dist-test-slave-wvz3"
server_key: "78e301e4b4c4009d7d98cdde0a98fed4"
server_key_iv: "83b9edcfd5dfedb4bd79bf0ef7603845"
server_key_version: "encryptionkey@0"
I20260620 07:06:36.518401 25511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:06:36.521991 25526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:36.523452 25511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:06:36.523618 25511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
format_stamp: "Formatted at 2026-06-20 07:06:36 on dist-test-slave-wvz3"
server_key: "78e301e4b4c4009d7d98cdde0a98fed4"
server_key_iv: "83b9edcfd5dfedb4bd79bf0ef7603845"
server_key_version: "encryptionkey@0"
I20260620 07:06:36.523804 25511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:36.554462 25511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:36.558764 25511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:36.559042 25511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:36.560061 25511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:36.561410 25511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:36.561491 25511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:36.561620 25511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:36.561681 25511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:36.591526 25511 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39859
I20260620 07:06:36.591555 25638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39859 every 8 connection(s)
I20260620 07:06:36.592845 25511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:06:36.597636 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25511
I20260620 07:06:36.597854 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:06:36.598739 11591 external_mini_cluster.cc:1428] Setting key 52c92bce9eee2ab757b2e7f420b2d4fe
Jun 20 07:06:36 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939196, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:36.615178 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:36.595379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:47063 (local address 127.11.81.254:43651)
0620 07:06:36.595784 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:36.595789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:36.596824 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 07:06:36.599348 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:36.599354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:36.599444 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:36.599572 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:36.601561 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:36.602253 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:36.604421 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:36.604805 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:36.604899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:36.608978 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:36.608984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:36.608987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:36.609035 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:36.612154 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:36.612914 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:36.612919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:36.612923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:36.613064 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:36.613432 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:36.613436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:36.613439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:36.613769 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:36.613904 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:36.614593 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 07:06:36.614823 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":194,"threads_started":1}
I20260620 07:06:36.616086 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:36.595954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:47063)
0620 07:06:36.596660 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:36.596705 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:36.598999 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:36.599835 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:36.599848 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:36.600479 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:36.601359 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:36.601386 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:36.602427 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:36.602432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:36.604283 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:36.604299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:36.605018 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:36.605025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:36.605312 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:36.606063 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:36.606091 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:36.608744 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:36.612312 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:36.612320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:36.612336 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:36.612791 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:36.613152 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:36.613155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:36.613158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:36.613329 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:36.613869 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:36.613875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:36.614229 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 07:06:36.614457 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 07:06:36.614885 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260620 07:06:36.617677 25639 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:36.617969 25639 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:36.618709 25639 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:36.621212 25450 ts_manager.cc:194] Registered new tserver with Master: ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859)
I20260620 07:06:36.623397 25450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:47063
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:06:36.652599 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:36.973644 25645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:36.973939 25645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:36.974062 25645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:36.981295 25645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:36.981423 25645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:36.981593 25645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:06:36.991267 25645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:36.992666 25645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:36.993999 25645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:37.008131 25650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:37.008147 25651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:37.008098 25653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:37.009332 25645 server_base.cc:1034] running on GCE node
I20260620 07:06:37.010002 25645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:37.011049 25645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:37.012271 25645 hybrid_clock.cc:648] HybridClock initialized: now 1781939197012202 us; error 61 us; skew 500 ppm
I20260620 07:06:37.012621 25645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:37 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939197, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:37.016003 25645 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:06:37.017530 25645 webserver.cc:460] Webserver started at http://127.11.81.194:38415/ using document root <none> and password file <none>
I20260620 07:06:37.018232 25645 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:37.018347 25645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:37.018635 25645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:37.021750 25645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "d1c9b659df1342b8baa7038d36d21fd1"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "1dd23ca78e8060afdcbba5030dfaf0f8"
server_key_iv: "59c437ef19be8a39aea13c14b3711eb6"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.022567 25645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "d1c9b659df1342b8baa7038d36d21fd1"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "1dd23ca78e8060afdcbba5030dfaf0f8"
server_key_iv: "59c437ef19be8a39aea13c14b3711eb6"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.027951 25645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:06:37.031456 25660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.032969 25645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:06:37.033139 25645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "d1c9b659df1342b8baa7038d36d21fd1"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "1dd23ca78e8060afdcbba5030dfaf0f8"
server_key_iv: "59c437ef19be8a39aea13c14b3711eb6"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.033340 25645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:37.047134 25645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:37.051383 25645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:37.051658 25645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:37.052615 25645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:37.053965 25645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:37.054041 25645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.054153 25645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:37.054207 25645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.085244 25645 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:45323
I20260620 07:06:37.085279 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:45323 every 8 connection(s)
I20260620 07:06:37.086513 25645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:06:37.094607 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25645
I20260620 07:06:37.094807 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:06:37.095341 11591 external_mini_cluster.cc:1428] Setting key 37f8168da4aa4a85f6918f2927d0dad2
Jun 20 07:06:37 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939197, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:37.107327 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.088990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:39577 (local address 127.11.81.254:43651)
0620 07:06:37.089170 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.089175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.090375 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 07:06:37.092274 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.092279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.092358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.092467 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:37.093994 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.094716 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.096480 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.096957 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.097098 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.101414 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:37.101422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:37.101428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:37.101468 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:37.103992 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.104819 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.104823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.104825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.104939 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.105410 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.105414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.105415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.105690 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:37.105805 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.106638 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.106883 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:06:37.108209 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.089417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:39577)
0620 07:06:37.090159 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.090214 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.091975 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.092692 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.092700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.093215 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:37.093803 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.093820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.094889 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.094895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.096339 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.096353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.097187 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.097193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.097491 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.098295 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:37.098322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:37.101120 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:37.104134 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:37.104144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:37.104163 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:37.104714 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:37.105049 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:37.105054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:37.105057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:37.105311 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:37.105805 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:37.105815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:37.106268 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.106502 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.106958 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":287,"threads_started":1}
I20260620 07:06:37.109663 25773 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:37.109956 25773 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:37.110709 25773 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:37.112246 25450 ts_manager.cc:194] Registered new tserver with Master: d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323)
I20260620 07:06:37.113076 25450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:39577
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:06:37.147746 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:37.471380 25778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:37.471690 25778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:37.471798 25778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:37.480082 25778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:37.480226 25778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:37.480407 25778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:06:37.491375 25778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:37.492688 25778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:37.493976 25778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:37.506886 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:37.506899 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:37.506932 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:37.507498 25778 server_base.cc:1034] running on GCE node
I20260620 07:06:37.508249 25778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:37.509243 25778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:37.510439 25778 hybrid_clock.cc:648] HybridClock initialized: now 1781939197510389 us; error 65 us; skew 500 ppm
I20260620 07:06:37.510769 25778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:37 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939197, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:37.513955 25778 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:06:37.515621 25778 webserver.cc:460] Webserver started at http://127.11.81.195:45483/ using document root <none> and password file <none>
I20260620 07:06:37.516341 25778 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:37.516464 25778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:37.516760 25778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:06:37.519783 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "5e67ffb5815a4fffa469a8c19571ec61"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "f0a36a21f43e3436f9c595265d7e50ff"
server_key_iv: "00abe073dbf25adb49123536781e9949"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.520577 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "5e67ffb5815a4fffa469a8c19571ec61"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "f0a36a21f43e3436f9c595265d7e50ff"
server_key_iv: "00abe073dbf25adb49123536781e9949"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.526028 25778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:06:37.529645 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.530989 25778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:06:37.531145 25778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "5e67ffb5815a4fffa469a8c19571ec61"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "f0a36a21f43e3436f9c595265d7e50ff"
server_key_iv: "00abe073dbf25adb49123536781e9949"
server_key_version: "encryptionkey@0"
I20260620 07:06:37.531376 25778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:37.564939 25778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:37.569056 25778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:37.569343 25778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:37.570341 25778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:37.571826 25778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:37.571900 25778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.571990 25778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:37.572034 25778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:37.603065 25778 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33669
I20260620 07:06:37.603080 25905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33669 every 8 connection(s)
I20260620 07:06:37.604367 25778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:06:37.610263 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25778
I20260620 07:06:37.610479 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:06:37.611009 11591 external_mini_cluster.cc:1428] Setting key da89400bde141e1cd3efbf0c77547ad5
Jun 20 07:06:37 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939197, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:37.626009 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.607022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:41931 (local address 127.11.81.254:43651)
0620 07:06:37.607227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.607233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.608552 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 07:06:37.610534 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.610540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.610654 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.610744 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:37.612763 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.613859 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.615724 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.616152 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.616261 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.619919 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:37.619924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:37.619927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:37.619965 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:37.622503 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.623440 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.623444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.623446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.623605 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.624031 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.624034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.624036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.624312 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:37.624430 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.625428 (+ 998us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.625677 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:06:37.626754 25639 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:37.627408 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.607556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:41931)
0620 07:06:37.608364 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.608410 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.610179 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.611063 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.611075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.611776 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:37.612581 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.612609 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.614098 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.614103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.615585 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.615599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.616404 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.616410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.616632 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.617297 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:37.617321 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:37.619667 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:37.622688 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:37.622696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:37.622712 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:37.623261 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:37.623702 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:37.623707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:37.623709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:37.623914 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:37.624463 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:37.624470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:37.625006 (+ 536us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.625320 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.625851 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":374,"threads_started":1}
I20260620 07:06:37.628691 25906 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:37.628993 25906 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:37.629734 25906 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:37.631234 25450 ts_manager.cc:194] Registered new tserver with Master: 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195:33669)
I20260620 07:06:37.632064 25450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:41931
I20260620 07:06:37.640239 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:06:37.657728 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.644847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.11.81.254:43651)
0620 07:06:37.645013 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.645019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.645461 (+ 442us) server_negotiation.cc:407] Connection header received
0620 07:06:37.645606 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.645610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.645680 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.645755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:37.647145 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.647942 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.649330 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.649772 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.649880 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.652440 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:37.652445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:37.652449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:37.652487 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:37.655062 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.655845 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.655850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.655852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.656056 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:37.656449 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:37.656452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:37.656454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:37.656806 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:37.657041 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.657294 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.657485 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:06:37.662213 25450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:06:37.669090 25450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:06:37.698683 25918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.682253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45323 (local address 127.0.0.1:46528)
0620 07:06:37.684574 (+ 2321us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.684622 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.684803 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.687253 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.687262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.687317 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:37.687753 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.687772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.691242 (+ 3470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.691249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.695448 (+ 4199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.695465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.697592 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.697598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.697862 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.697930 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.698090 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.698311 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2053,"thread_start_us":1914,"threads_started":1}
I20260620 07:06:37.699405 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.682736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.11.81.194:45323)
0620 07:06:37.684009 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.684020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.685007 (+ 987us) server_negotiation.cc:407] Connection header received
0620 07:06:37.686445 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.686455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.686762 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.686963 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:37.687963 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.691048 (+ 3085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.695639 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.697155 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.697291 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.697529 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.698175 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.698371 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":1009,"threads_started":1}
I20260620 07:06:37.703922 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.682423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.11.81.195:33669)
0620 07:06:37.696083 (+ 13660us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.696092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.696130 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:06:37.696232 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.696241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.696554 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.696763 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:37.697377 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.699234 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.701776 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.702877 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.702998 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.703227 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.703465 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.703620 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13477,"thread_start_us":13319,"threads_started":1}
I20260620 07:06:37.703979 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.682342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33669 (local address 127.0.0.1:55034)
0620 07:06:37.694081 (+ 11739us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.694105 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.694252 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.696743 (+ 2491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.696748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.696774 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:37.697154 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.697168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.699454 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.699458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.701592 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.701608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.703070 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.703076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.703245 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.703304 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.703397 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.703533 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11558,"mutex_wait_us":1,"thread_start_us":13875,"threads_started":1}
I20260620 07:06:37.703724 25708 tablet_service.cc:1467] Processing CreateTablet for tablet fd320813f3e54c34a66843f6215b4820 (DEFAULT_TABLE table=test-table [id=7d6e1ce3967649dcac8c823797a2a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:37.704739 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.685092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59648)
0620 07:06:37.696437 (+ 11345us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.696456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.696596 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.697260 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.697265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.697293 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:37.697704 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.697715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.699869 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.699874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.701815 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.701829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.703948 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.703954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.704132 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.704160 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.704242 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.704368 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11184,"mutex_wait_us":25,"thread_start_us":8629,"threads_started":1}
I20260620 07:06:37.705686 25708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd320813f3e54c34a66843f6215b4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:37.707795 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.685583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.11.81.193:39859)
0620 07:06:37.695526 (+ 9943us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.695537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.696574 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 07:06:37.696753 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.696761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.697032 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.697254 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:37.697879 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.699707 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.701985 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.703437 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.703579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.703793 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.707415 (+ 3622us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.707560 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9726,"thread_start_us":9523,"threads_started":1}
I20260620 07:06:37.707976 25841 tablet_service.cc:1467] Processing CreateTablet for tablet fd320813f3e54c34a66843f6215b4820 (DEFAULT_TABLE table=test-table [id=7d6e1ce3967649dcac8c823797a2a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:37.709465 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd320813f3e54c34a66843f6215b4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:37.710716 25574 tablet_service.cc:1467] Processing CreateTablet for tablet fd320813f3e54c34a66843f6215b4820 (DEFAULT_TABLE table=test-table [id=7d6e1ce3967649dcac8c823797a2a26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:37.711840 25574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd320813f3e54c34a66843f6215b4820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:37.720618 25924 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Bootstrap starting.
I20260620 07:06:37.723909 25925 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Bootstrap starting.
I20260620 07:06:37.724715 25924 tablet_bootstrap.cc:654] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:37.725409 25926 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Bootstrap starting.
I20260620 07:06:37.726193 25924 log.cc:824] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:37.727576 25925 tablet_bootstrap.cc:654] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:37.728821 25926 tablet_bootstrap.cc:654] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:37.728816 25925 log.cc:824] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:37.729118 25924 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: No bootstrap required, opened a new log
I20260620 07:06:37.729383 25924 ts_tablet_manager.cc:1397] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:06:37.729904 25926 log.cc:824] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:37.731837 25925 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: No bootstrap required, opened a new log
I20260620 07:06:37.732093 25925 ts_tablet_manager.cc:1397] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 07:06:37.732271 25926 tablet_bootstrap.cc:492] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: No bootstrap required, opened a new log
I20260620 07:06:37.732515 25926 ts_tablet_manager.cc:1397] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:06:37.739711 25924 raft_consensus.cc:348] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.739899 25926 raft_consensus.cc:348] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.740203 25924 raft_consensus.cc:374] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:37.740227 25926 raft_consensus.cc:374] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:37.740306 25926 raft_consensus.cc:729] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6c6f4194fb4488b3bf9e0032c4c477, State: Initialized, Role: FOLLOWER
I20260620 07:06:37.740300 25924 raft_consensus.cc:729] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c9b659df1342b8baa7038d36d21fd1, State: Initialized, Role: FOLLOWER
I20260620 07:06:37.740872 25926 consensus_queue.cc:260] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.740976 25924 consensus_queue.cc:260] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.742781 25926 ts_tablet_manager.cc:1428] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260620 07:06:37.742501 25925 raft_consensus.cc:348] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.742945 25925 raft_consensus.cc:374] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:37.743039 25925 raft_consensus.cc:729] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e67ffb5815a4fffa469a8c19571ec61, State: Initialized, Role: FOLLOWER
I20260620 07:06:37.743808 25925 consensus_queue.cc:260] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.743971 25773 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:37.744578 25924 ts_tablet_manager.cc:1428] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260620 07:06:37.745517 25906 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:37.746052 25925 ts_tablet_manager.cc:1428] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
W20260620 07:06:37.838428 25774 tablet.cc:2369] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:06:37.845122 25640 tablet.cc:2369] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:06:37.855933 25907 tablet.cc:2369] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:06:37.872391 25932 raft_consensus.cc:482] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:06:37.872717 25932 raft_consensus.cc:504] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.874393 25932 leader_election.cc:288] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323), ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859)
I20260620 07:06:37.880326 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.874702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45323 (local address 127.11.81.195:51363)
0620 07:06:37.874900 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.874922 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.875054 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.875592 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.875597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.875627 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:37.875844 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.875864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.877121 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.877126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.878453 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.878465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.879608 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.879613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.879788 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.879813 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.879893 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.880042 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 07:06:37.880326 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.874857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:51363 (local address 127.11.81.194:45323)
0620 07:06:37.875068 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.875074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.875096 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:06:37.875175 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.875180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.875259 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.875383 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:37.876050 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.877008 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.878616 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.879467 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.879607 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.879685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.879941 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.880070 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:06:37.881201 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.875580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:50607 (local address 127.11.81.193:39859)
0620 07:06:37.876008 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.876014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.876205 (+ 191us) server_negotiation.cc:407] Connection header received
0620 07:06:37.876436 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.876441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.876517 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.876612 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:37.877091 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.877789 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.879455 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.880239 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.880385 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.880451 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.880908 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.881013 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260620 07:06:37.881320 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.875122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.11.81.195:50607)
0620 07:06:37.876116 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:37.876133 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:37.876246 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:37.876623 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:37.876626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:37.876645 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:37.876923 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.876933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.877943 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.877947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.879308 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:37.879324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.880593 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.880599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:37.880755 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.880780 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:06:37.880862 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:06:37.880990 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":676,"threads_started":1}
I20260620 07:06:37.881399 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd320813f3e54c34a66843f6215b4820" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c9b659df1342b8baa7038d36d21fd1" is_pre_election: true
I20260620 07:06:37.882009 25728 raft_consensus.cc:2454] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 0.
I20260620 07:06:37.882393 25594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd320813f3e54c34a66843f6215b4820" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" is_pre_election: true
I20260620 07:06:37.882645 25794 leader_election.cc:302] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e67ffb5815a4fffa469a8c19571ec61, d1c9b659df1342b8baa7038d36d21fd1; no voters:
I20260620 07:06:37.882920 25594 raft_consensus.cc:2454] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 0.
I20260620 07:06:37.883025 25932 raft_consensus.cc:2781] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:06:37.883172 25932 raft_consensus.cc:482] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:06:37.883270 25932 raft_consensus.cc:3037] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:37.885500 25932 raft_consensus.cc:504] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.886060 25932 leader_election.cc:288] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 election: Requested vote from peers d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323), ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859)
I20260620 07:06:37.886525 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd320813f3e54c34a66843f6215b4820" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c9b659df1342b8baa7038d36d21fd1"
I20260620 07:06:37.886564 25594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd320813f3e54c34a66843f6215b4820" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
I20260620 07:06:37.886711 25728 raft_consensus.cc:3037] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:37.886744 25594 raft_consensus.cc:3037] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:37.888899 25594 raft_consensus.cc:2454] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 1.
I20260620 07:06:37.888926 25728 raft_consensus.cc:2454] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 1.
I20260620 07:06:37.889281 25794 leader_election.cc:302] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e67ffb5815a4fffa469a8c19571ec61, d1c9b659df1342b8baa7038d36d21fd1; no voters:
I20260620 07:06:37.889590 25932 raft_consensus.cc:2781] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:37.890043 25932 raft_consensus.cc:686] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 LEADER]: Becoming Leader. State: Replica: 5e67ffb5815a4fffa469a8c19571ec61, State: Running, Role: LEADER
I20260620 07:06:37.890512 25932 consensus_queue.cc:237] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } }
I20260620 07:06:37.894972 25449 catalog_manager.cc:5495] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "5e67ffb5815a4fffa469a8c19571ec61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20260620 07:06:37.935524 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.930127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.11.81.195:33669)
0620 07:06:37.930337 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.930343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:37.930364 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:06:37.930461 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:37.930465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:37.930543 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:37.930700 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:06:37.931095 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.931850 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.933313 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:37.933717 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:37.933829 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:37.934392 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:06:37.934918 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:06:37.935092 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:37.935141 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:06:37.935224 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:06:37.943982 25594 raft_consensus.cc:1261] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Refusing update from remote peer 5e67ffb5815a4fffa469a8c19571ec61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:37.944094 25728 raft_consensus.cc:1261] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Refusing update from remote peer 5e67ffb5815a4fffa469a8c19571ec61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:37.944931 25932 consensus_queue.cc:1035] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:37.945240 25934 consensus_queue.cc:1035] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:37.954921 25937 mvcc.cc:205] Tried to move back new op lower bound from 7298822954767933440 to 7298822954563411968. Current Snapshot: MvccSnapshot[applied={T|T < 7298822954767933440}]
I20260620 07:06:37.958235 25938 mvcc.cc:205] Tried to move back new op lower bound from 7298822954767933440 to 7298822954563411968. Current Snapshot: MvccSnapshot[applied={T|T < 7298822954767933440}]
I20260620 07:06:37.960179 25939 mvcc.cc:205] Tried to move back new op lower bound from 7298822954767933440 to 7298822954563411968. Current Snapshot: MvccSnapshot[applied={T|T < 7298822954767933440}]
I20260620 07:06:37.980656 25449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:37.980955 25449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:37.984859 25449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Sending DeleteTablet for 3 replicas of tablet fd320813f3e54c34a66843f6215b4820
I20260620 07:06:37.986121 25841 tablet_service.cc:1514] Processing DeleteTablet for tablet fd320813f3e54c34a66843f6215b4820 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:37 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:55034
I20260620 07:06:37.986210 25708 tablet_service.cc:1514] Processing DeleteTablet for tablet fd320813f3e54c34a66843f6215b4820 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:37 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46528
I20260620 07:06:37.986215 25574 tablet_service.cc:1514] Processing DeleteTablet for tablet fd320813f3e54c34a66843f6215b4820 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:37 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:59648
I20260620 07:06:37.987555 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25511
I20260620 07:06:38.002105 25949 tablet_replica.cc:330] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: stopping tablet replica
I20260620 07:06:38.002676 25949 raft_consensus.cc:2229] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:38.003265 25949 raft_consensus.cc:2258] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:38.010934 25948 tablet_replica.cc:330] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: stopping tablet replica
I20260620 07:06:38.011497 25948 raft_consensus.cc:2229] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:38.012089 25948 raft_consensus.cc:2258] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:38.012981 25947 tablet_replica.cc:330] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: stopping tablet replica
I20260620 07:06:38.013459 25947 raft_consensus.cc:2229] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:06:38.014218 25947 raft_consensus.cc:2258] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:38.015062 25949 ts_tablet_manager.cc:1905] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:38.016355 25947 ts_tablet_manager.cc:1905] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:06:38 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:06:38.020689 25948 ts_tablet_manager.cc:1905] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:38.022250 25949 ts_tablet_manager.cc:1918] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:38.022442 25949 log.cc:1196] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/fd320813f3e54c34a66843f6215b4820
I20260620 07:06:38.023134 25949 ts_tablet_manager.cc:1939] T fd320813f3e54c34a66843f6215b4820 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting consensus metadata
I20260620 07:06:38.025231 25434 catalog_manager.cc:4841] TS ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859): tablet fd320813f3e54c34a66843f6215b4820 (table test-table [id=7d6e1ce3967649dcac8c823797a2a26d]) successfully deleted
I20260620 07:06:38.028692 25947 ts_tablet_manager.cc:1918] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:38.029110 25947 log.cc:1196] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/fd320813f3e54c34a66843f6215b4820
I20260620 07:06:38.029445 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:37.988463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.11.81.193:39859)
0620 07:06:37.988657 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:37.988663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:38.004790 (+ 16127us) server_negotiation.cc:407] Connection header received
0620 07:06:38.005102 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:38.005108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:38.005198 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:38.005272 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:38.007336 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.008463 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.012940 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.013556 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.013670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:38.021220 (+ 7550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:38.021238 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:38.021256 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:38.021309 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:38.025187 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.026296 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.026306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.026309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.026490 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.027044 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.027048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.027051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.027623 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:38.027977 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:38.028597 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 07:06:38.028961 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:06:38.029645 25947 ts_tablet_manager.cc:1939] T fd320813f3e54c34a66843f6215b4820 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting consensus metadata
I20260620 07:06:38.030404 25948 ts_tablet_manager.cc:1918] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:38.030776 25948 log.cc:1196] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/fd320813f3e54c34a66843f6215b4820
I20260620 07:06:38.031188 25634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:38.031212 25948 ts_tablet_manager.cc:1939] T fd320813f3e54c34a66843f6215b4820 P d1c9b659df1342b8baa7038d36d21fd1: Deleting consensus metadata
I20260620 07:06:38.033311 25434 catalog_manager.cc:4841] TS d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323): tablet fd320813f3e54c34a66843f6215b4820 (table test-table [id=7d6e1ce3967649dcac8c823797a2a26d]) successfully deleted
I20260620 07:06:38.033974 25437 catalog_manager.cc:4841] TS 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195:33669): tablet fd320813f3e54c34a66843f6215b4820 (table test-table [id=7d6e1ce3967649dcac8c823797a2a26d]) successfully deleted
I20260620 07:06:38.122474 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25511
I20260620 07:06:38.137610 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25645
Jun 20 07:06:38 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:06:38.154861 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:38.138499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.11.81.194:45323)
0620 07:06:38.138653 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:38.138657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:38.138676 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:06:38.138838 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:38.138842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:38.138911 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:38.138991 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:38.140663 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.141461 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.143361 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.143755 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.143852 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:38.148949 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:38.148967 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:38.148984 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:38.149027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:38.152110 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.152917 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.152924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.152927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.153048 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.153449 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.153461 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.153463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.153851 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:38.154084 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:38.154384 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 07:06:38.154566 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:06:38.155606 25768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:38.243680 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25645
I20260620 07:06:38.258157 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25778
Jun 20 07:06:38 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:06:38.276865 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:38.259087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.11.81.195:33669)
0620 07:06:38.259254 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:38.259258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:38.259279 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:06:38.259403 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:38.259407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:38.259495 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:38.259573 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:38.261442 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.262780 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.264596 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:38.265051 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:38.265162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:38.270145 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:38.270158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:38.270174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:38.270217 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:38.273337 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.274330 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.274339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.274342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.274541 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:38.275072 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:38.275076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:38.275078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:38.275624 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:38.275878 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:38.276345 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 07:06:38.276529 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:06:38.277801 25901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:38.398316 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25778
I20260620 07:06:38.414053 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25418
I20260620 07:06:38.415191 25480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:38.548624 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25418
I20260620 07:06:38.566188 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--webserver_interface=127.11.81.254
--webserver_port=42017
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:38.890798 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:38.891099 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:38.891211 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:38.898625 25955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:38.898747 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:38.898797 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:38.898885 25955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:38.898943 25955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:38.909018 25955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:38.910373 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:38.911717 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:38.933202 25961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:38.933271 25964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:38.933236 25962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:38.934123 25955 server_base.cc:1034] running on GCE node
I20260620 07:06:38.934947 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:38.936622 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:38.937851 25955 hybrid_clock.cc:648] HybridClock initialized: now 1781939198937795 us; error 42 us; skew 500 ppm
I20260620 07:06:38.938304 25955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:38 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939198, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:38.941934 25955 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:38.943609 25955 webserver.cc:460] Webserver started at http://127.11.81.254:42017/ using document root <none> and password file <none>
I20260620 07:06:38.944340 25955 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:38.944453 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:38.950459 25955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:06:38.953492 25971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:38.954885 25955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:06:38.955030 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b80dc0794f444f4eab2f1e438287cbfb"
format_stamp: "Formatted at 2026-06-20 07:06:35 on dist-test-slave-wvz3"
server_key: "13d285dbcf709126dc020e684d7fe22c"
server_key_iv: "0f9a878349b8801ecce09ff229ea182c"
server_key_version: "encryptionkey@0"
I20260620 07:06:38.956175 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:38.971168 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:38.975636 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:38.975885 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:38.994506 25955 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43651
I20260620 07:06:38.994539 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43651 every 8 connection(s)
I20260620 07:06:38.996187 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:38.998890 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 25955
I20260620 07:06:38.999899 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:39859
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=33243
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:39.005959 26023 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:06:39.010429 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap starting.
I20260620 07:06:39.031253 26023 log.cc:824] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:39.044749 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:06:39.045360 26023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap complete.
I20260620 07:06:39.054817 26023 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:39.055795 26023 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Initialized, Role: FOLLOWER
I20260620 07:06:39.056493 26023 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:39.056731 26023 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:39.056854 26023 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:39.057062 26023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:06:39.061262 26023 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:39.061813 26023 leader_election.cc:302] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80dc0794f444f4eab2f1e438287cbfb; no voters:
I20260620 07:06:39.062436 26023 leader_election.cc:288] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:06:39.062600 26027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:06:39.063329 26027 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 LEADER]: Becoming Leader. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Running, Role: LEADER
I20260620 07:06:39.064018 26027 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:39.064591 26023 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:39.066661 26029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:39.067171 26029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:39.067109 26028 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80dc0794f444f4eab2f1e438287cbfb. Latest consensus state: current_term: 2 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:39.067412 26028 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:39.075989 26034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:39.086123 26034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:39.087628 26034 catalog_manager.cc:1259] Loaded cluster ID: 8a437bb2b703475c94b4d53c7aaaef32
I20260620 07:06:39.087734 26034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:06:39.090795 26034 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 07:06:39.112257 26034 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 07:06:39.132095 26034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:06:39.133900 26034 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Loaded TSK: 0
I20260620 07:06:39.135542 26034 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:06:39.358392 26025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:39.358697 26025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:39.358834 26025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:39.366513 26025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:39.366644 26025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:39.366792 26025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:06:39.376741 26025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:39859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=33243
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:39.378093 26025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:39.379469 26025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:39.394122 26051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:39.394229 26053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:39.394918 26050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:39.395994 26025 server_base.cc:1034] running on GCE node
I20260620 07:06:39.396631 26025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:39.397691 26025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:39.398903 26025 hybrid_clock.cc:648] HybridClock initialized: now 1781939199398844 us; error 55 us; skew 500 ppm
I20260620 07:06:39.399231 26025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:39 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939199, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:39.406764 26025 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:06:39.408435 26025 webserver.cc:460] Webserver started at http://127.11.81.193:33243/ using document root <none> and password file <none>
I20260620 07:06:39.409153 26025 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:39.409287 26025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:39.415079 26025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 07:06:39.418143 26060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.419744 26025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:06:39.419903 26025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
format_stamp: "Formatted at 2026-06-20 07:06:36 on dist-test-slave-wvz3"
server_key: "78e301e4b4c4009d7d98cdde0a98fed4"
server_key_iv: "83b9edcfd5dfedb4bd79bf0ef7603845"
server_key_version: "encryptionkey@0"
I20260620 07:06:39.420917 26025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:39.432466 26025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:39.437651 26025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:39.438014 26025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:39.439196 26025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:39.440893 26025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:39.441008 26025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.441130 26025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:39.441206 26025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.474972 26025 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39859
I20260620 07:06:39.475006 26172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39859 every 8 connection(s)
I20260620 07:06:39.476583 26025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:06:39.486110 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26025
I20260620 07:06:39.486963 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:45323
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=38415
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:39 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939199, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:39.502585 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:39.480089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:58055)
0620 07:06:39.480935 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:39.480982 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:39.483356 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:39.485117 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:39.485129 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:39.485899 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:39.486703 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:39.486732 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.488513 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.488517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:39.489763 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:39.489776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.490836 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.490843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:39.491123 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:39.491828 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:39.491850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:39.494178 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:39.498399 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:39.498416 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:39.498430 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:39.498935 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:39.499464 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:39.499467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:39.499470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:39.499683 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:39.500574 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:39.500580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:39.500890 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 07:06:39.501139 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:06:39.501574 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":311,"threads_started":1}
I20260620 07:06:39.502879 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:39.479705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:58055 (local address 127.11.81.254:43651)
0620 07:06:39.480272 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:39.480286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:39.481140 (+ 854us) server_negotiation.cc:407] Connection header received
0620 07:06:39.483957 (+ 2817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:39.483976 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:39.484510 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:39.484954 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:39.486998 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.488340 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.489913 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.490592 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.490794 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:39.494468 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:39.494478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:39.494499 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:39.494544 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:39.498207 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:39.499074 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:39.499083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:39.499091 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:39.499348 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:39.499782 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:39.499787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:39.499790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:39.500429 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:39.500593 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:39.501366 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 07:06:39.501790 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260620 07:06:39.505640 26173 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:39.505934 26173 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:39.506678 26173 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:39.508852 25988 ts_manager.cc:194] Registered new tserver with Master: ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859)
I20260620 07:06:39.510360 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:58055
W20260620 07:06:39.812872 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:39.813198 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:39.813328 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:39.820904 26177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:39.821041 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:39.821197 26177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:06:39.831153 26177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:45323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=38415
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:39.832502 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:39.833801 26177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:39.847221 26182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:39.847266 26185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:39.847186 26183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:39.848181 26177 server_base.cc:1034] running on GCE node
I20260620 07:06:39.848742 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:39.849664 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:39.850869 26177 hybrid_clock.cc:648] HybridClock initialized: now 1781939199850821 us; error 35 us; skew 500 ppm
I20260620 07:06:39.851181 26177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:39 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939199, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:39.854404 26177 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:06:39.855839 26177 webserver.cc:460] Webserver started at http://127.11.81.194:38415/ using document root <none> and password file <none>
I20260620 07:06:39.856510 26177 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:39.856619 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:39.862073 26177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 07:06:39.864681 26192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.866250 26177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:06:39.866387 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "d1c9b659df1342b8baa7038d36d21fd1"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "1dd23ca78e8060afdcbba5030dfaf0f8"
server_key_iv: "59c437ef19be8a39aea13c14b3711eb6"
server_key_version: "encryptionkey@0"
I20260620 07:06:39.867359 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:39.895344 26177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:39.899951 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:39.900203 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:39.901181 26177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:39.902527 26177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:39.902621 26177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.902701 26177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:39.902752 26177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:39.932961 26177 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:45323
I20260620 07:06:39.933027 26304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:45323 every 8 connection(s)
I20260620 07:06:39.934332 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:06:39.943138 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26177
I20260620 07:06:39.943872 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:33669
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=45483
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:06:39 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939199, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:39.955682 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:39.936781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:41141 (local address 127.11.81.254:43651)
0620 07:06:39.936944 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:39.936948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:39.938073 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 07:06:39.939955 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:39.939961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:39.940059 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:39.940145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:39.941748 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.942502 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.944813 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.945310 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.945419 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:39.949672 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:39.949677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:39.949681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:39.949719 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:39.952599 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:39.953376 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:39.953381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:39.953385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:39.953531 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:39.953909 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:39.953913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:39.953916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:39.954258 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:39.954389 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:39.955069 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 07:06:39.955335 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:06:39.956607 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:39.937210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:41141)
0620 07:06:39.937902 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:39.937941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:39.939674 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:39.940434 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:39.940442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:39.940968 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:39.941571 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:39.941589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.942661 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.942666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:39.944615 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:39.944631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:39.945537 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:39.945552 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:39.945825 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:39.946577 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:39.946605 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:39.949415 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:39.952747 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:39.952755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:39.952770 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:39.953256 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:39.953625 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:39.953628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:39.953631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:39.953807 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:39.954375 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:39.954381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:39.954711 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 07:06:39.954942 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:06:39.955403 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":266,"threads_started":1}
I20260620 07:06:39.958037 26305 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:39.958384 26305 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:39.959223 26305 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:39.960834 25988 ts_manager.cc:194] Registered new tserver with Master: d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323)
I20260620 07:06:39.961740 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:41141
W20260620 07:06:40.277334 26308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:40.277616 26308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:40.277742 26308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:40.285143 26308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:40.285266 26308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:40.285415 26308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:06:40.295329 26308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=45483
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:40.296638 26308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:40.297931 26308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:40.311167 26314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:40.311136 26313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:40.311215 26316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:40.312381 26308 server_base.cc:1034] running on GCE node
I20260620 07:06:40.312975 26308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:40.313962 26308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:40.315151 26308 hybrid_clock.cc:648] HybridClock initialized: now 1781939200315104 us; error 41 us; skew 500 ppm
I20260620 07:06:40.315475 26308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:40 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939200, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:40.318548 26308 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:06:40.320115 26308 webserver.cc:460] Webserver started at http://127.11.81.195:45483/ using document root <none> and password file <none>
I20260620 07:06:40.320788 26308 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:40.320917 26308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:40.326740 26308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:06:40.329629 26323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:40.331085 26308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:06:40.331231 26308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "5e67ffb5815a4fffa469a8c19571ec61"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "f0a36a21f43e3436f9c595265d7e50ff"
server_key_iv: "00abe073dbf25adb49123536781e9949"
server_key_version: "encryptionkey@0"
I20260620 07:06:40.332264 26308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:40.357407 26308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:40.362248 26308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:40.362509 26308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:40.363536 26308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:40.364850 26308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:40.364936 26308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:40.365017 26308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:40.365069 26308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:40.395764 26308 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33669
I20260620 07:06:40.395835 26435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33669 every 8 connection(s)
I20260620 07:06:40.397197 26308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:06:40.399662 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26308
Jun 20 07:06:40 dist-test-slave-wvz3 krb5kdc[25405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939200, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:06:40.415532 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.400553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.11.81.254:43651)
0620 07:06:40.402293 (+ 1740us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.402298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.402315 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:06:40.402374 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.402378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.402437 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.402536 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:40.404146 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.404897 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.406221 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.406640 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.406748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.409574 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:40.409579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:40.409582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:40.409614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:40.412776 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.413511 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.413517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.413520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.413689 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.414066 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.414071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.414074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.414444 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:40.414734 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.414950 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.415159 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"spinlock_wait_cycles":11520,"thread_start_us":1526,"threads_started":1}
I20260620 07:06:40.418303 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.399736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33657 (local address 127.11.81.254:43651)
0620 07:06:40.399964 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.399970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.401433 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 07:06:40.403639 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.403644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.403718 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.403802 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:40.405325 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.406073 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.407630 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.408052 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.408156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.411838 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:40.411844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:40.411849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:40.411909 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:40.414943 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.415822 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.415827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.415830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.415962 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.416453 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.416457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.416460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.416816 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:40.416938 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.417796 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.417971 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 07:06:40.419528 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.400333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:33657)
0620 07:06:40.401243 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.401300 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.403374 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.404069 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.404079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.404600 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:06:40.405165 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.405182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.406219 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.406223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.407502 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.407514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.408262 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.408270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.408497 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.409092 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:06:40.409113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:06:40.411604 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:06:40.415088 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:40.415099 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:40.415118 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:40.415696 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:40.416075 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:06:40.416079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:06:40.416082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:06:40.416332 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:06:40.416946 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:06:40.416963 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:06:40.417391 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.417691 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.418269 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":335,"threads_started":1}
I20260620 07:06:40.420796 26436 heartbeater.cc:347] Connected to a master server at 127.11.81.254:43651
I20260620 07:06:40.421092 26436 heartbeater.cc:464] Registering TS with master...
I20260620 07:06:40.421797 26436 heartbeater.cc:511] Master 127.11.81.254:43651 requested a full tablet report, sending...
I20260620 07:06:40.423218 25988 ts_manager.cc:194] Registered new tserver with Master: 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195:33669)
I20260620 07:06:40.424060 25988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:33657
I20260620 07:06:40.430838 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:06:40.437963 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.431759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.11.81.254:43651)
0620 07:06:40.431938 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.431945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.432377 (+ 432us) server_negotiation.cc:407] Connection header received
0620 07:06:40.432596 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.432600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.432779 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.432923 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:06:40.433426 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.434224 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.435845 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.436311 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.436427 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.436918 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:06:40.437425 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:06:40.437548 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.437636 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.437718 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:06:40.438830 25988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:06:40.446163 25988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:06:40.469085 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.458708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33669 (local address 127.0.0.1:55046)
0620 07:06:40.461163 (+ 2455us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.461180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.461299 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.461963 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.461968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.461989 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:40.462331 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.462342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.464548 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.464553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.466297 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.466309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.468061 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.468067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.468274 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.468334 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.468457 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.468654 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"mutex_wait_us":34,"thread_start_us":2425,"threads_started":1}
I20260620 07:06:40.469518 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.458112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59670)
0620 07:06:40.458943 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.458997 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.460305 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.461553 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.461564 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.461597 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:40.461974 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.461986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.463833 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.463838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.466493 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.466508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.468649 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.468655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.468817 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.468848 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.468946 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.469096 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":111,"thread_start_us":363,"threads_started":1}
I20260620 07:06:40.469933 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.458346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.11.81.193:39859)
0620 07:06:40.460873 (+ 2527us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.460881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.460912 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:06:40.461019 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.461028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.461304 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.462194 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:40.462424 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.463696 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.466646 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.468419 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.468545 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.468743 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.468994 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.469438 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2369,"thread_start_us":2241,"threads_started":1}
I20260620 07:06:40.470114 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.459307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.11.81.195:33669)
0620 07:06:40.460577 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.460587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.461325 (+ 738us) server_negotiation.cc:407] Connection header received
0620 07:06:40.461429 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.461437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.461685 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.461875 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:40.462512 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.464388 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.466450 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.467849 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.469353 (+ 1504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.469563 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.469636 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.469800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":933,"threads_started":1}
I20260620 07:06:40.470115 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.459602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.11.81.194:45323)
0620 07:06:40.460171 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.460181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.461869 (+ 1688us) server_negotiation.cc:407] Connection header received
0620 07:06:40.462093 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.462100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.462282 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.462433 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:40.463306 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.464772 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.467108 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.468532 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.468662 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.468851 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.469690 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.469852 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260620 07:06:40.470813 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.459710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45323 (local address 127.0.0.1:46542)
0620 07:06:40.461741 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.461760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.461876 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.462711 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.462715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.462739 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:40.463104 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.463117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.464905 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.464909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.466960 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.466971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.469016 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.469026 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.469187 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.469527 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.470302 (+ 775us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.470450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"mutex_wait_us":27,"thread_start_us":1764,"threads_started":1}
I20260620 07:06:40.473863 26371 tablet_service.cc:1467] Processing CreateTablet for tablet 9edaaf5d92e14286b49447f90614c368 (DEFAULT_TABLE table=test-table [id=90f89cf089764ff6a28559daabd400b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:40.473837 26240 tablet_service.cc:1467] Processing CreateTablet for tablet 9edaaf5d92e14286b49447f90614c368 (DEFAULT_TABLE table=test-table [id=90f89cf089764ff6a28559daabd400b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:40.473863 26108 tablet_service.cc:1467] Processing CreateTablet for tablet 9edaaf5d92e14286b49447f90614c368 (DEFAULT_TABLE table=test-table [id=90f89cf089764ff6a28559daabd400b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:06:40.475270 26108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edaaf5d92e14286b49447f90614c368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:40.475261 26371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edaaf5d92e14286b49447f90614c368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:40.475262 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9edaaf5d92e14286b49447f90614c368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:06:40.490818 26452 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Bootstrap starting.
I20260620 07:06:40.492967 26454 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Bootstrap starting.
I20260620 07:06:40.496011 26453 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Bootstrap starting.
I20260620 07:06:40.496397 26454 tablet_bootstrap.cc:654] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:40.496397 26452 tablet_bootstrap.cc:654] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:40.497709 26452 log.cc:824] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:40.497709 26454 log.cc:824] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:40.499271 26453 tablet_bootstrap.cc:654] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Neither blocks nor log segments found. Creating new log.
I20260620 07:06:40.500478 26453 log.cc:824] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:40.501116 26452 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: No bootstrap required, opened a new log
I20260620 07:06:40.501439 26452 ts_tablet_manager.cc:1397] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260620 07:06:40.501849 26454 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: No bootstrap required, opened a new log
I20260620 07:06:40.502112 26454 ts_tablet_manager.cc:1397] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 07:06:40.503124 26453 tablet_bootstrap.cc:492] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: No bootstrap required, opened a new log
I20260620 07:06:40.503397 26453 ts_tablet_manager.cc:1397] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:06:40.510718 26453 raft_consensus.cc:348] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.511132 26453 raft_consensus.cc:374] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:40.511229 26453 raft_consensus.cc:729] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6c6f4194fb4488b3bf9e0032c4c477, State: Initialized, Role: FOLLOWER
I20260620 07:06:40.511904 26453 consensus_queue.cc:260] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.512787 26454 raft_consensus.cc:348] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.512789 26452 raft_consensus.cc:348] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.513242 26454 raft_consensus.cc:374] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:40.513242 26452 raft_consensus.cc:374] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:06:40.513334 26454 raft_consensus.cc:729] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e67ffb5815a4fffa469a8c19571ec61, State: Initialized, Role: FOLLOWER
I20260620 07:06:40.513334 26452 raft_consensus.cc:729] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c9b659df1342b8baa7038d36d21fd1, State: Initialized, Role: FOLLOWER
I20260620 07:06:40.513475 26173 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:40.514082 26454 consensus_queue.cc:260] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.514082 26452 consensus_queue.cc:260] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.521718 26453 ts_tablet_manager.cc:1428] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260620 07:06:40.522311 26452 ts_tablet_manager.cc:1428] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Time spent starting tablet: real 0.021s user 0.020s sys 0.001s
I20260620 07:06:40.522884 26305 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:40.522893 26436 heartbeater.cc:503] Master 127.11.81.254:43651 was elected leader, sending a full tablet report...
I20260620 07:06:40.523638 26454 ts_tablet_manager.cc:1428] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Time spent starting tablet: real 0.021s user 0.014s sys 0.009s
I20260620 07:06:40.530212 26459 raft_consensus.cc:482] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:06:40.530503 26459 raft_consensus.cc:504] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.532377 26459 leader_election.cc:288] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859), d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323)
I20260620 07:06:40.538275 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.532843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:55517 (local address 127.11.81.193:39859)
0620 07:06:40.533055 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.533061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.533080 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:06:40.533153 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.533158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.533240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.533337 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:40.533893 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.534601 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.536331 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.537422 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.537536 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.537587 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.537963 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.538067 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:06:40.538355 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.532593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.11.81.195:55517)
0620 07:06:40.532833 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.532849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.532966 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.533544 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.533549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.533592 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:40.533751 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.533760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.534743 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.534747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.536176 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.536187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.537576 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.537582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.537772 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.537809 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.537919 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.538064 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2}
I20260620 07:06:40.539316 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.532997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:45323 (local address 127.11.81.195:58399)
0620 07:06:40.533676 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.533689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:06:40.533807 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:06:40.534084 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:06:40.534087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:06:40.534109 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:06:40.534391 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.534401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.535339 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.535343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.536988 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:06:40.537002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.538659 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.538666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:06:40.538828 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.538859 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:06:40.538959 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:06:40.539051 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":408,"threads_started":1}
I20260620 07:06:40.539433 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.533209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:58399 (local address 127.11.81.194:45323)
0620 07:06:40.533402 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.533409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.533771 (+ 362us) server_negotiation.cc:407] Connection header received
0620 07:06:40.533926 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.533930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.533987 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.534056 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:06:40.534523 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.535186 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.537158 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.538281 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.538430 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.538499 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.539027 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.539168 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:06:40.539536 26128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edaaf5d92e14286b49447f90614c368" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" is_pre_election: true
I20260620 07:06:40.540184 26128 raft_consensus.cc:2454] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 0.
I20260620 07:06:40.540658 26260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edaaf5d92e14286b49447f90614c368" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c9b659df1342b8baa7038d36d21fd1" is_pre_election: true
I20260620 07:06:40.540887 26324 leader_election.cc:302] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e67ffb5815a4fffa469a8c19571ec61, ab6c6f4194fb4488b3bf9e0032c4c477; no voters:
I20260620 07:06:40.541074 26260 raft_consensus.cc:2454] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 0.
I20260620 07:06:40.541322 26459 raft_consensus.cc:2781] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:06:40.541462 26459 raft_consensus.cc:482] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:06:40.541559 26459 raft_consensus.cc:3037] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:40.543948 26459 raft_consensus.cc:504] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.544533 26459 leader_election.cc:288] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 election: Requested vote from peers ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859), d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323)
I20260620 07:06:40.545035 26128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edaaf5d92e14286b49447f90614c368" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
I20260620 07:06:40.545077 26260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9edaaf5d92e14286b49447f90614c368" candidate_uuid: "5e67ffb5815a4fffa469a8c19571ec61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c9b659df1342b8baa7038d36d21fd1"
I20260620 07:06:40.545205 26128 raft_consensus.cc:3037] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:40.545291 26260 raft_consensus.cc:3037] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:06:40.547380 26128 raft_consensus.cc:2454] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 1.
I20260620 07:06:40.547480 26260 raft_consensus.cc:2454] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67ffb5815a4fffa469a8c19571ec61 in term 1.
I20260620 07:06:40.548038 26324 leader_election.cc:302] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e67ffb5815a4fffa469a8c19571ec61, ab6c6f4194fb4488b3bf9e0032c4c477; no voters:
I20260620 07:06:40.548316 26459 raft_consensus.cc:2781] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:06:40.548758 26459 raft_consensus.cc:686] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 LEADER]: Becoming Leader. State: Replica: 5e67ffb5815a4fffa469a8c19571ec61, State: Running, Role: LEADER
I20260620 07:06:40.549209 26459 consensus_queue.cc:237] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } }
I20260620 07:06:40.553884 25986 catalog_manager.cc:5495] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "5e67ffb5815a4fffa469a8c19571ec61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e67ffb5815a4fffa469a8c19571ec61" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33669 } health_report { overall_health: HEALTHY } } }
I20260620 07:06:40.580826 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.575222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.11.81.195:33669)
0620 07:06:40.575410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.575414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.575490 (+ 76us) server_negotiation.cc:407] Connection header received
0620 07:06:40.575678 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.575682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.575771 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.575911 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:06:40.576276 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.576993 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.578546 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.579017 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.579119 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.579669 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:06:40.580275 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:06:40.580390 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.580541 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.580615 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:06:40.589696 26128 raft_consensus.cc:1261] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Refusing update from remote peer 5e67ffb5815a4fffa469a8c19571ec61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:40.589696 26260 raft_consensus.cc:1261] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Refusing update from remote peer 5e67ffb5815a4fffa469a8c19571ec61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:06:40.590785 26459 consensus_queue.cc:1035] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c9b659df1342b8baa7038d36d21fd1" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:40.591166 26462 consensus_queue.cc:1035] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab6c6f4194fb4488b3bf9e0032c4c477" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:06:40.601615 26464 mvcc.cc:205] Tried to move back new op lower bound from 7298822965604118528 to 7298822965453336576. Current Snapshot: MvccSnapshot[applied={T|T < 7298822965604118528}]
I20260620 07:06:40.602716 26466 mvcc.cc:205] Tried to move back new op lower bound from 7298822965604118528 to 7298822965453336576. Current Snapshot: MvccSnapshot[applied={T|T < 7298822965604118528}]
I20260620 07:06:40.618868 25987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:40.619206 25987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:06:40.623096 25987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Sending DeleteTablet for 3 replicas of tablet 9edaaf5d92e14286b49447f90614c368
I20260620 07:06:40.624498 26240 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edaaf5d92e14286b49447f90614c368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:40 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:46542
I20260620 07:06:40.624675 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26025
I20260620 07:06:40.624747 26108 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edaaf5d92e14286b49447f90614c368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:40 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:59670
I20260620 07:06:40.625777 26371 tablet_service.cc:1514] Processing DeleteTablet for tablet 9edaaf5d92e14286b49447f90614c368 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:06:40 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:55046
I20260620 07:06:40.626596 26476 tablet_replica.cc:330] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: stopping tablet replica
I20260620 07:06:40.629094 26476 raft_consensus.cc:2229] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:40.629741 26476 raft_consensus.cc:2258] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:40.634867 26477 tablet_replica.cc:330] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: stopping tablet replica
I20260620 07:06:40.635263 26477 raft_consensus.cc:2229] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:06:40.636947 26477 raft_consensus.cc:2258] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:40.638300 26476 ts_tablet_manager.cc:1905] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:40.643112 26475 tablet_replica.cc:330] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: stopping tablet replica
I20260620 07:06:40.643697 26475 raft_consensus.cc:2229] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:06:40.644240 26475 raft_consensus.cc:2258] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:06:40.645457 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.625438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.11.81.193:39859)
0620 07:06:40.626374 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.626379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.626407 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:06:40.626518 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.626534 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.626632 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.626750 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:40.628592 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.629888 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.631822 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.632503 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.632634 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.636257 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:40.636274 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:40.636292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:40.636339 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:40.641475 (+ 5136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.642880 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.642890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.642893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.643072 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.643542 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.643547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.643551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.644198 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:40.644574 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.644855 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.645106 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814}
I20260620 07:06:40.646447 26168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:40.646838 26477 ts_tablet_manager.cc:1905] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:40.647686 26475 ts_tablet_manager.cc:1905] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:40.655105 26475 ts_tablet_manager.cc:1918] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:40.655418 26475 log.cc:1196] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/9edaaf5d92e14286b49447f90614c368
I20260620 07:06:40.656095 26475 ts_tablet_manager.cc:1939] T 9edaaf5d92e14286b49447f90614c368 P d1c9b659df1342b8baa7038d36d21fd1: Deleting consensus metadata
I20260620 07:06:40.656523 26477 ts_tablet_manager.cc:1918] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:40.656749 26477 log.cc:1196] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/9edaaf5d92e14286b49447f90614c368
I20260620 07:06:40.657346 26477 ts_tablet_manager.cc:1939] T 9edaaf5d92e14286b49447f90614c368 P 5e67ffb5815a4fffa469a8c19571ec61: Deleting consensus metadata
I20260620 07:06:40.658495 25972 catalog_manager.cc:4841] TS d1c9b659df1342b8baa7038d36d21fd1 (127.11.81.194:45323): tablet 9edaaf5d92e14286b49447f90614c368 (table test-table [id=90f89cf089764ff6a28559daabd400b9]) successfully deleted
I20260620 07:06:40.659191 25975 catalog_manager.cc:4841] TS 5e67ffb5815a4fffa469a8c19571ec61 (127.11.81.195:33669): tablet 9edaaf5d92e14286b49447f90614c368 (table test-table [id=90f89cf089764ff6a28559daabd400b9]) successfully deleted
I20260620 07:06:40.768498 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26025
I20260620 07:06:40.783546 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26177
W20260620 07:06:40.783756 25972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:39859 (error 108)
I20260620 07:06:40.785950 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.785452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59682)
0620 07:06:40.785638 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.785768 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 07:06:40.786237 25972 catalog_manager.cc:4584] TS ab6c6f4194fb4488b3bf9e0032c4c477 (127.11.81.193:39859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9edaaf5d92e14286b49447f90614c368: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
I20260620 07:06:40.797832 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.784562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.11.81.194:45323)
0620 07:06:40.784746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.784752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.784772 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:06:40.784838 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.784844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.784928 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.785043 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:40.786605 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.787433 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.788742 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.789148 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.789264 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.792006 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:40.792024 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:40.792040 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:40.792078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:40.794906 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.795797 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.795805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.795808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.795942 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.796299 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.796304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.796307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.796684 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:40.796956 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.797289 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.797504 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:06:40.798588 26300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:40.840051 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.839465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59694)
0620 07:06:40.839708 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.839846 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:06:40.841225 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.840828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59696)
0620 07:06:40.840986 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.841076 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:06:40.886885 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26177
I20260620 07:06:40.901330 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26308
I20260620 07:06:40.902122 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.901625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59700)
0620 07:06:40.901844 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.901930 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:06:40.903362 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.902878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59714)
0620 07:06:40.903112 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.903196 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 07:06:40.917964 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.902225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.11.81.195:33669)
0620 07:06:40.902411 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:40.902417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:40.902439 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:06:40.902687 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:40.902693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:40.902786 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:40.902889 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:40.904414 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.905653 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.906920 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:40.907628 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:40.907790 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:40.910584 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:40.910599 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:40.910634 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:40.910685 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:40.914588 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.915478 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.915488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.915492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.915679 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:40.916133 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:40.916138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:40.916141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:40.916691 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:40.917001 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:40.917270 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:06:40.917516 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:06:40.918781 26431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:40.984107 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.983555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59716)
0620 07:06:40.983783 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.983907 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:06:40.985255 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:40.984798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:39859 (local address 127.0.0.1:59728)
0620 07:06:40.984993 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:40.985095 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:06:41.049590 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26308
I20260620 07:06:41.065778 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 25955
I20260620 07:06:41.066946 26018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:41.180045 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 25955
I20260620 07:06:41.198289 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--webserver_interface=127.11.81.254
--webserver_port=42017
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:06:41.530236 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:41.530568 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:41.530710 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:41.538226 26483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:06:41.538360 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:41.538415 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:41.538513 26483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:06:41.538686 26483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:06:41.549818 26483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=42017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:41.551234 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:41.552592 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:41.564400 26489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:41.564405 26491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:41.564618 26488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:41.565227 26483 server_base.cc:1034] running on GCE node
I20260620 07:06:41.566023 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:41.567556 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:41.568786 26483 hybrid_clock.cc:648] HybridClock initialized: now 1781939201568728 us; error 48 us; skew 500 ppm
I20260620 07:06:41.569164 26483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:41 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939201, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:41.572328 26483 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:06:41.573887 26483 webserver.cc:460] Webserver started at http://127.11.81.254:42017/ using document root <none> and password file <none>
I20260620 07:06:41.574551 26483 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:41.574651 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:41.580474 26483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:06:41.583519 26498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:41.584955 26483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:06:41.585111 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b80dc0794f444f4eab2f1e438287cbfb"
format_stamp: "Formatted at 2026-06-20 07:06:35 on dist-test-slave-wvz3"
server_key: "13d285dbcf709126dc020e684d7fe22c"
server_key_iv: "0f9a878349b8801ecce09ff229ea182c"
server_key_version: "encryptionkey@0"
I20260620 07:06:41.586083 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:41.629961 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:41.634277 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:41.634506 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:41.652385 26483 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43651
I20260620 07:06:41.652403 26549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43651 every 8 connection(s)
I20260620 07:06:41.653924 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:06:41.660902 26550 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:06:41.661312 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26483
I20260620 07:06:41.662099 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:39859
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=33243
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:41.664433 26550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap starting.
I20260620 07:06:41.690783 26550 log.cc:824] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:06:41.712646 26550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:06:41.713207 26550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb: Bootstrap complete.
I20260620 07:06:41.722327 26550 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:41.723316 26550 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Initialized, Role: FOLLOWER
I20260620 07:06:41.724013 26550 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:41.724252 26550 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:06:41.724362 26550 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:06:41.724524 26550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:06:41.728629 26550 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:41.729141 26550 leader_election.cc:302] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80dc0794f444f4eab2f1e438287cbfb; no voters:
I20260620 07:06:41.729765 26550 leader_election.cc:288] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:06:41.729984 26554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:06:41.731076 26554 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [term 3 LEADER]: Becoming Leader. State: Replica: b80dc0794f444f4eab2f1e438287cbfb, State: Running, Role: LEADER
I20260620 07:06:41.731684 26554 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } }
I20260620 07:06:41.731909 26550 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:06:41.734418 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80dc0794f444f4eab2f1e438287cbfb. Latest consensus state: current_term: 3 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:41.734444 26555 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b80dc0794f444f4eab2f1e438287cbfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80dc0794f444f4eab2f1e438287cbfb" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43651 } } }
I20260620 07:06:41.734879 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:41.734884 26555 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80dc0794f444f4eab2f1e438287cbfb [sys.catalog]: This master's current role is: LEADER
I20260620 07:06:41.735858 26561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:06:41.754360 26561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:06:41.755921 26561 catalog_manager.cc:1259] Loaded cluster ID: 8a437bb2b703475c94b4d53c7aaaef32
I20260620 07:06:41.756044 26561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 07:06:41.758041 26561 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f79f909c9ad google::LogMessage::Fail() at ??:0
@ 0x7f79f909f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f79f909c4de google::LogMessage::Flush() at ??:0
@ 0x7f79f909da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7a0eacf512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7a0eacc976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7a0eb54610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7a0eb54439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f79fbce8345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f79fbceb91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f79fbceb799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f79fbcc6eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f79fefa06db start_thread at ??:0
@ 0x7f79f7f3661f clone at ??:0
W20260620 07:06:42.056516 26552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:42.056824 26552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:42.056953 26552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:42.065167 26552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:42.065305 26552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:42.065471 26552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:06:42.075538 26552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:39859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=33243
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:42.076814 26552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:42.078161 26552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:42.091471 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.091430 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.091491 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:42.092257 26552 server_base.cc:1034] running on GCE node
I20260620 07:06:42.092872 26552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:42.093832 26552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:42.095031 26552 hybrid_clock.cc:648] HybridClock initialized: now 1781939202094993 us; error 32 us; skew 500 ppm
I20260620 07:06:42.095402 26552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:42 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939202, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:42.098434 26552 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:06:42.099858 26552 webserver.cc:460] Webserver started at http://127.11.81.193:33243/ using document root <none> and password file <none>
I20260620 07:06:42.100525 26552 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:42.100643 26552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:42.105952 26552 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:06:42.109120 26601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:42.110522 26552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 07:06:42.110692 26552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "ab6c6f4194fb4488b3bf9e0032c4c477"
format_stamp: "Formatted at 2026-06-20 07:06:36 on dist-test-slave-wvz3"
server_key: "78e301e4b4c4009d7d98cdde0a98fed4"
server_key_iv: "83b9edcfd5dfedb4bd79bf0ef7603845"
server_key_version: "encryptionkey@0"
I20260620 07:06:42.111738 26552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:42.126381 26552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:42.130661 26552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:42.130926 26552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:42.131927 26552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:42.133734 26608 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 07:06:42.139231 26608 ts_tablet_manager.cc:1852] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 07:06:42.140312 26608 ts_tablet_manager.cc:1905] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:06:42.148036 26608 ts_tablet_manager.cc:1918] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:06:42.148232 26608 log.cc:1196] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/9edaaf5d92e14286b49447f90614c368
I20260620 07:06:42.148695 26608 ts_tablet_manager.cc:1939] T 9edaaf5d92e14286b49447f90614c368 P ab6c6f4194fb4488b3bf9e0032c4c477: Deleting consensus metadata
I20260620 07:06:42.149106 26552 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 07:06:42.149235 26552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 07:06:42.149360 26552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:42.149435 26552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:42.182606 26552 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:39859
I20260620 07:06:42.182655 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:39859 every 8 connection(s)
I20260620 07:06:42.184072 26552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:06:42.189186 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.187142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:56599)
0620 07:06:42.187927 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.188264 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":299,"threads_started":1}
I20260620 07:06:42.189713 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26552
W20260620 07:06:42.189857 26715 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43651 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:43651: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
I20260620 07:06:42.190387 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:45323
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=38415
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:42.190797 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.190414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:47761)
0620 07:06:42.190572 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.190666 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:06:42.191722 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.191376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:50599)
0620 07:06:42.191509 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.191593 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 07:06:42.191928 26715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:06:42.516165 26718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:42.516469 26718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:42.516606 26718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:42.524058 26718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:42.524178 26718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:42.524329 26718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:06:42.534474 26718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:45323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=38415
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:42.535856 26718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:42.537180 26718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:42.550078 26723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.550020 26724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.550045 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:42.550894 26718 server_base.cc:1034] running on GCE node
I20260620 07:06:42.551604 26718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:42.552591 26718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:42.553797 26718 hybrid_clock.cc:648] HybridClock initialized: now 1781939202553749 us; error 40 us; skew 500 ppm
I20260620 07:06:42.554126 26718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:42 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939202, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:42.557500 26718 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:06:42.559053 26718 webserver.cc:460] Webserver started at http://127.11.81.194:38415/ using document root <none> and password file <none>
I20260620 07:06:42.559808 26718 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:42.559932 26718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:42.565428 26718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:06:42.568248 26733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:42.569662 26718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:06:42.569828 26718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "d1c9b659df1342b8baa7038d36d21fd1"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "1dd23ca78e8060afdcbba5030dfaf0f8"
server_key_iv: "59c437ef19be8a39aea13c14b3711eb6"
server_key_version: "encryptionkey@0"
I20260620 07:06:42.570909 26718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:42.597810 26718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:42.601907 26718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:42.602186 26718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:42.603272 26718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:42.604660 26718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:42.604754 26718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:42.604858 26718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:42.604926 26718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:42.635852 26718 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:45323
I20260620 07:06:42.635931 26845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:45323 every 8 connection(s)
I20260620 07:06:42.637295 26718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:06:42.642141 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.640217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:60677)
0620 07:06:42.640924 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.641180 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
W20260620 07:06:42.642910 26846 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43651 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:43651: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
I20260620 07:06:42.643630 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26718
I20260620 07:06:42.643916 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.643508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:48355)
0620 07:06:42.643667 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.643750 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:06:42.644261 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:33669
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:43651
--webserver_port=45483
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:06:42.644773 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:42.644469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:47883)
0620 07:06:42.644584 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:42.644657 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 07:06:42.645048 26846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:06:42.966352 26849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:06:42.966638 26849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:06:42.966732 26849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:06:42.973980 26849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:06:42.974098 26849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:06:42.974251 26849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:06:42.983975 26849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=45483
--tserver_master_addrs=127.11.81.254:43651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:06:42.985229 26849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:06:42.986618 26849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:06:42.999326 26855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.999352 26857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:06:42.999333 26854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:06:43.000154 26849 server_base.cc:1034] running on GCE node
I20260620 07:06:43.000797 26849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:06:43.001775 26849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:06:43.002981 26849 hybrid_clock.cc:648] HybridClock initialized: now 1781939203002918 us; error 56 us; skew 500 ppm
I20260620 07:06:43.003334 26849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:06:43 dist-test-slave-wvz3 krb5kdc[25405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939203, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:06:43.006526 26849 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:06:43.008070 26849 webserver.cc:460] Webserver started at http://127.11.81.195:45483/ using document root <none> and password file <none>
I20260620 07:06:43.008767 26849 fs_manager.cc:359] Metadata directory not provided
I20260620 07:06:43.008884 26849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:06:43.014559 26849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:06:43.017469 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:43.018862 26849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:06:43.019057 26849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "5e67ffb5815a4fffa469a8c19571ec61"
format_stamp: "Formatted at 2026-06-20 07:06:37 on dist-test-slave-wvz3"
server_key: "f0a36a21f43e3436f9c595265d7e50ff"
server_key_iv: "00abe073dbf25adb49123536781e9949"
server_key_version: "encryptionkey@0"
I20260620 07:06:43.020076 26849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:06:43.033679 26849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:06:43.037719 26849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:06:43.038002 26849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:06:43.038987 26849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:06:43.040416 26849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:06:43.040527 26849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:43.040630 26849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:06:43.040697 26849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:06:43.070773 26849 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33669
I20260620 07:06:43.070842 26976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33669 every 8 connection(s)
I20260620 07:06:43.072273 26849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:06:43.075354 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 26849
I20260620 07:06:43.077420 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:43.075374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:49355)
0620 07:06:43.076227 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:43.076478 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":331,"threads_started":1}
W20260620 07:06:43.078070 26977 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:43651 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:43651: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
I20260620 07:06:43.078816 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:43.078526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:38555)
0620 07:06:43.078646 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:43.078710 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:06:43.079764 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:43.079415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:40029)
0620 07:06:43.079564 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:43.079646 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 07:06:43.079994 26977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:06:43.193460 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:43.192578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:45255)
0620 07:06:43.193139 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:43.193240 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":236,"threads_started":1}
I20260620 07:06:43.646531 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:43.645680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:59347)
0620 07:06:43.646186 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:43.646285 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260620 07:06:44.081329 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:44.080549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:58565)
0620 07:06:44.081022 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:44.081128 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260620 07:06:44.195374 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:44.194531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:59235)
0620 07:06:44.195064 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:44.195164 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":269,"threads_started":1}
I20260620 07:06:44.648329 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:44.647582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:55753)
0620 07:06:44.648067 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:44.648160 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":231,"threads_started":1}
I20260620 07:06:45.082908 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:45.082235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:41045)
0620 07:06:45.082673 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:45.082754 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":207,"threads_started":1}
I20260620 07:06:45.197199 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:45.196364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:33793)
0620 07:06:45.196872 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:45.197019 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":240,"threads_started":1}
I20260620 07:06:45.649961 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:45.649240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:53297)
0620 07:06:45.649688 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:45.649781 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260620 07:06:46.084607 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:46.083811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:54917)
0620 07:06:46.084340 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:46.084420 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":305,"threads_started":1}
I20260620 07:06:46.198997 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:46.198237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:51629)
0620 07:06:46.198734 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:46.198819 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260620 07:06:46.651718 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:46.650905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:45179)
0620 07:06:46.651398 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:46.651491 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260620 07:06:47.086253 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:47.085494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:49871)
0620 07:06:47.085980 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:47.086063 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
I20260620 07:06:47.200827 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:47.200023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:58641)
0620 07:06:47.200548 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:47.200651 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":251,"threads_started":1}
I20260620 07:06:47.653419 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:47.652642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:34763)
0620 07:06:47.653142 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:47.653236 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260620 07:06:48.088529 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:48.087353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:60277)
0620 07:06:48.088225 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:48.088312 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":637,"threads_started":1}
I20260620 07:06:48.202607 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:48.201832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:47073)
0620 07:06:48.202342 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:48.202439 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260620 07:06:48.655155 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:48.654438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:33113)
0620 07:06:48.654888 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:48.654985 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260620 07:06:49.090587 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:49.089599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:42819)
0620 07:06:49.090313 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:49.090411 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":372,"threads_started":1}
I20260620 07:06:49.204460 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:49.203616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:34281)
0620 07:06:49.204166 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:49.204250 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":240,"threads_started":1}
I20260620 07:06:49.660048 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:49.656266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:59683)
0620 07:06:49.659734 (+ 3468us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:49.659831 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3345,"thread_start_us":3156,"threads_started":1}
I20260620 07:06:50.092523 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:50.091617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:33735)
0620 07:06:50.092189 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:50.092295 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":253,"threads_started":1}
I20260620 07:06:50.206194 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:50.205474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:59653)
0620 07:06:50.205931 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:50.206026 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260620 07:06:50.661765 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:50.661005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:34825)
0620 07:06:50.661503 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:50.661599 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":254,"threads_started":1}
I20260620 07:06:51.094453 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:51.093600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:53057)
0620 07:06:51.094150 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:51.094275 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":258,"threads_started":1}
I20260620 07:06:51.208140 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:51.207243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:40127)
0620 07:06:51.207807 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:51.207925 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":242,"threads_started":1}
I20260620 07:06:51.663445 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:51.662674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:35593)
0620 07:06:51.663145 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:51.663232 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260620 07:06:52.096366 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:52.095544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:41287)
0620 07:06:52.096075 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:52.096168 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":267,"threads_started":1}
I20260620 07:06:52.210027 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:52.209243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:49883)
0620 07:06:52.209776 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:52.209856 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":315,"threads_started":1}
I20260620 07:06:52.665093 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:52.664340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:42215)
0620 07:06:52.664824 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:52.664926 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260620 07:06:53.098258 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:53.097418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:38805)
0620 07:06:53.097980 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:53.098082 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
I20260620 07:06:53.211966 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:53.211095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:37501)
0620 07:06:53.211642 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:53.211738 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260620 07:06:53.666874 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:53.666069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:46171)
0620 07:06:53.666579 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:53.666688 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":245,"threads_started":1}
I20260620 07:06:54.100243 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:54.099433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:42485)
0620 07:06:54.099966 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:54.100062 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":231,"threads_started":1}
I20260620 07:06:54.213851 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:54.213018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:47949)
0620 07:06:54.213524 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:54.213623 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260620 07:06:54.668598 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:54.667885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:59593)
0620 07:06:54.668331 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:54.668418 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260620 07:06:55.102106 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:55.101324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:39505)
0620 07:06:55.101825 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:55.101933 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260620 07:06:55.215858 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:55.214976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:34493)
0620 07:06:55.215550 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:55.215674 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":307,"threads_started":1}
I20260620 07:06:55.670261 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:55.669462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:37473)
0620 07:06:55.669975 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:55.670074 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":234,"threads_started":1}
I20260620 07:06:56.104010 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:56.103108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:38685)
0620 07:06:56.103668 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:56.103786 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":324,"threads_started":1}
I20260620 07:06:56.217530 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:56.216770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:51623)
0620 07:06:56.217232 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:56.217345 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":224,"threads_started":1}
I20260620 07:06:56.672279 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:56.671397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:38815)
0620 07:06:56.671878 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:56.672031 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260620 07:06:57.105715 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:57.104916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:53285)
0620 07:06:57.105417 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:57.105517 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260620 07:06:57.219652 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:57.218821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.193:57137)
0620 07:06:57.219363 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:57.219466 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260620 07:06:57.674388 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:57.673546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.194:48585)
0620 07:06:57.674078 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:57.674192 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
I20260620 07:06:58.082051 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26552
I20260620 07:06:58.106115 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:58.083131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.11.81.193:39859)
0620 07:06:58.083741 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:58.083815 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:58.083863 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:06:58.085953 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:58.085964 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:58.086438 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:58.086776 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:58.088342 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.089547 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.090865 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.091569 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.091739 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:58.099894 (+ 8155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:58.099903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:58.099921 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:58.099971 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:58.102915 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.103664 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.103672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.103679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.103863 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.104258 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.104262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.104265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.104867 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:58.105121 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:58.105464 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:06:58.105809 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260620 07:06:58.107681 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:58.106909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:43651 (local address 127.11.81.195:49035)
0620 07:06:58.107422 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 07:06:58.107514 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:43651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260620 07:06:58.107730 26710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:58.193769 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26552
I20260620 07:06:58.208446 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26718
I20260620 07:06:58.227164 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:58.209420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.11.81.194:45323)
0620 07:06:58.210020 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:58.210033 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:58.210073 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:06:58.211775 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:58.211786 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:58.212232 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:58.212532 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:58.213922 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.215271 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.216729 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.217496 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.217653 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:58.220423 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:58.220431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:58.220448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:58.220494 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:58.223690 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.224429 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.224439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.224448 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.224629 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.225041 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.225046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.225050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.225667 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:58.225905 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:58.226296 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 07:06:58.226788 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":254,"threads_started":1}
I20260620 07:06:58.228631 26841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:58.312630 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26718
I20260620 07:06:58.326654 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 26849
I20260620 07:06:58.344596 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:06:58.327903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.11.81.195:33669)
0620 07:06:58.328723 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 07:06:58.328742 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:06:58.328792 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:06:58.330703 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:06:58.330712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:06:58.331148 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:06:58.331490 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:06:58.332823 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.334026 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.335257 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:06:58.335960 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:06:58.336091 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:06:58.338683 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:06:58.338690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:06:58.338706 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:06:58.338747 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:06:58.341615 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.342257 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.342263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.342269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.342440 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:06:58.342767 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:06:58.342770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:06:58.342772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:06:58.343264 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:06:58.343545 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 07:06:58.343869 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 07:06:58.344266 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":429,"threads_started":1}
I20260620 07:06:58.345999 26972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:06:58.428828 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 26849
2026-06-20T07:06:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24137 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:06:58 dist-test-slave-wvz3 krb5kdc[27034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:06:58 dist-test-slave-wvz3 krb5kdc[27034](info): set up 2 sockets
Jun 20 07:06:58 dist-test-slave-wvz3 krb5kdc[27034](info): commencing operation
krb5kdc: starting...
W20260620 07:06:59.471747 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.009s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:06:59 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:06:59.706935 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:00.048883 27047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:00.049264 27047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:00.049407 27047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:00.056958 27047 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:07:00.057090 27047 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:07:00.057147 27047 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:07:00.057215 27047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:07:00.057292 27047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:00.057386 27047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:00.057456 27047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:07:00.057529 27047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:07:00.067979 27047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:00.069414 27047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:00.070772 27047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:00.082824 27052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:00.082892 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:00.082859 27053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:00.083722 27047 server_base.cc:1034] running on GCE node
I20260620 07:07:00.084592 27047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:00.086202 27047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:00.087404 27047 hybrid_clock.cc:648] HybridClock initialized: now 1781939220087330 us; error 38 us; skew 500 ppm
I20260620 07:07:00.087741 27047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:00 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939220, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:00.090889 27047 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:07:00.092437 27047 webserver.cc:460] Webserver started at http://127.11.81.254:34871/ using document root <none> and password file <none>
I20260620 07:07:00.093142 27047 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:00.093242 27047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:00.093556 27047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:07:00.096661 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "c0fac99e8ccd4513a990a0ac71e553d4"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "f6543201f9125fd1964d3046319b0529"
server_key_iv: "002037588ea6ba90e3fc9a32a3554056"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.097445 27047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "c0fac99e8ccd4513a990a0ac71e553d4"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "f6543201f9125fd1964d3046319b0529"
server_key_iv: "002037588ea6ba90e3fc9a32a3554056"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.103117 27047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:07:00.106928 27062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:00.108469 27047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:00.108651 27047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c0fac99e8ccd4513a990a0ac71e553d4"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "f6543201f9125fd1964d3046319b0529"
server_key_iv: "002037588ea6ba90e3fc9a32a3554056"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.108868 27047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:00.123114 27047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:00.127806 27047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:00.128074 27047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:00.146484 27047 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35801
I20260620 07:07:00.146463 27113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35801 every 8 connection(s)
I20260620 07:07:00.148025 27047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:07:00.150676 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27047
I20260620 07:07:00.150857 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:07:00.151546 11591 external_mini_cluster.cc:1428] Setting key dc7e182bd33875fbbc671a6c1bb12f03
I20260620 07:07:00.152416 27114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:07:00 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:00.167412 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap starting.
I20260620 07:07:00.170852 27114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:00.172185 27114 log.cc:824] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:00.175637 27114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: No bootstrap required, opened a new log
I20260620 07:07:00.175820 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:00.153560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.11.81.254:35801)
0620 07:07:00.154310 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:00.154323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:00.154366 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:07:00.156015 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:00.156039 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:00.156474 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:00.156833 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:00.158806 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.160306 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.162122 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.162831 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.162994 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:00.168032 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:00.168042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:00.168059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:00.168101 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:00.171190 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:00.172111 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:00.172118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:00.172125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:00.172368 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:00.172806 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:00.172811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:00.172814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:00.173641 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:00.173861 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:00.174306 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 07:07:00.174697 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":336,"threads_started":1}
I20260620 07:07:00.183598 27114 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:00.184013 27114 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:00.184118 27114 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Initialized, Role: FOLLOWER
I20260620 07:07:00.184684 27114 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:00.184904 27114 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:07:00.185027 27114 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:07:00.185180 27114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:00.187577 27114 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:00.188109 27114 leader_election.cc:302] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fac99e8ccd4513a990a0ac71e553d4; no voters:
I20260620 07:07:00.188686 27114 leader_election.cc:288] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:07:00.188807 27119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:07:00.189427 27119 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Running, Role: LEADER
I20260620 07:07:00.190003 27119 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:00.190716 27114 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:07:00.192770 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fac99e8ccd4513a990a0ac71e553d4. Latest consensus state: current_term: 1 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:00.193248 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:00.193593 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:00.195035 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:00.196331 27127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:07:00.201721 27127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:07:00.211040 27127 catalog_manager.cc:1347] Generated new cluster ID: 82f9bcef75c5409c92788bd629126eb7
I20260620 07:07:00.211170 27127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:07:00.224923 27127 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:07:00.226858 27127 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:07:00.236243 27127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Generated new TSK 0
I20260620 07:07:00.237473 27127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:07:00.299741 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:00.633351 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:00.633646 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:00.633787 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:00.642014 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:00.642144 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:00.642262 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:07:00.652385 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:00.653666 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:00.655025 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:00.668299 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:00.668323 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:00.668406 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:00.669108 27140 server_base.cc:1034] running on GCE node
I20260620 07:07:00.669749 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:00.670740 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:00.671957 27140 hybrid_clock.cc:648] HybridClock initialized: now 1781939220671906 us; error 39 us; skew 500 ppm
I20260620 07:07:00.672286 27140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:00 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939220, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:00.675465 27140 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:07:00.676986 27140 webserver.cc:460] Webserver started at http://127.11.81.193:38897/ using document root <none> and password file <none>
I20260620 07:07:00.677676 27140 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:00.677798 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:00.678162 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:07:00.681309 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "6e3afec0722af5181afb6c406413c5a9"
server_key_iv: "f965aae0c3f7c0b13b3c7b1b950b91b7"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.682118 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "6e3afec0722af5181afb6c406413c5a9"
server_key_iv: "f965aae0c3f7c0b13b3c7b1b950b91b7"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.687879 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:07:00.691659 27155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:00.693187 27140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:07:00.693347 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "6e3afec0722af5181afb6c406413c5a9"
server_key_iv: "f965aae0c3f7c0b13b3c7b1b950b91b7"
server_key_version: "encryptionkey@0"
I20260620 07:07:00.693542 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:00.717094 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:00.721284 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:00.721549 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:00.722520 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:00.723904 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:00.723982 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:00.724072 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:00.724114 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:00.754470 27140 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:46229
I20260620 07:07:00.754546 27267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:46229 every 8 connection(s)
I20260620 07:07:00.755832 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:07:00.756215 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27140
I20260620 07:07:00.756433 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:07:00.757225 11591 external_mini_cluster.cc:1428] Setting key 4410d4ea5800df3230d1466a4e39ef83
Jun 20 07:07:00 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939220, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:00.779134 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:00.758802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:43933 (local address 127.11.81.254:35801)
0620 07:07:00.759314 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:00.759321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:00.760462 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 07:07:00.762875 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:00.762879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:00.762958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:00.763052 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:00.765095 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.766007 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.768099 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.768580 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.768705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:00.772956 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:00.772962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:00.772966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:00.773005 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:00.775540 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:00.776521 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:00.776526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:00.776528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:00.776661 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:00.777174 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:00.777178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:00.777180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:00.777530 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:00.777653 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:00.778614 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 07:07:00.778859 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":229,"threads_started":1}
I20260620 07:07:00.780575 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:00.759451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:43933)
0620 07:07:00.760269 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:00.760325 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:00.762553 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:00.763355 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:00.763366 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:00.764111 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:00.764904 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:00.764935 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.766164 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.766169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:00.767970 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:00.767986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:00.768829 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:00.768836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:00.769074 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:00.769717 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:00.769746 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:00.772568 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:00.775694 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:00.775705 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:00.775724 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:00.776356 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:00.776756 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:00.776760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:00.776764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:00.777021 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:00.777656 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:00.777664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:00.778134 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 07:07:00.778464 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 07:07:00.779063 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":285,"threads_started":1}
I20260620 07:07:00.782332 27268 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:00.782650 27268 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:07:00.783455 27268 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:07:00.785899 27079 ts_manager.cc:194] Registered new tserver with Master: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229)
I20260620 07:07:00.787945 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:43933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:07:00.814424 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:01.143827 27274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:01.144135 27274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:01.144269 27274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:01.151902 27274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:01.152022 27274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:01.152182 27274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:07:01.162243 27274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:01.163620 27274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:01.164918 27274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:01.178467 27279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:01.178510 27280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:01.178696 27282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:01.179311 27274 server_base.cc:1034] running on GCE node
I20260620 07:07:01.179976 27274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:01.181047 27274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:01.182297 27274 hybrid_clock.cc:648] HybridClock initialized: now 1781939221182230 us; error 61 us; skew 500 ppm
I20260620 07:07:01.182644 27274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:01 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939221, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:01.185956 27274 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:07:01.187585 27274 webserver.cc:460] Webserver started at http://127.11.81.194:45621/ using document root <none> and password file <none>
I20260620 07:07:01.188279 27274 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:01.188400 27274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:01.188699 27274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:07:01.191951 27274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "fd23c0c8c31547b9854265d01255f23b"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "89e6407384684bd5af8e631d8761e87d"
server_key_iv: "bfc67097f4835690ef22d57c578a2f43"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.192706 27274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "fd23c0c8c31547b9854265d01255f23b"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "89e6407384684bd5af8e631d8761e87d"
server_key_iv: "bfc67097f4835690ef22d57c578a2f43"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.198125 27274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:07:01.202456 27289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.204207 27274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:01.204420 27274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "fd23c0c8c31547b9854265d01255f23b"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "89e6407384684bd5af8e631d8761e87d"
server_key_iv: "bfc67097f4835690ef22d57c578a2f43"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.204635 27274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:01.229213 27274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:01.233947 27274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:01.234236 27274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:01.235246 27274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:01.236783 27274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:01.236866 27274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.236989 27274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:01.237041 27274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.268530 27274 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:43129
I20260620 07:07:01.268591 27401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:43129 every 8 connection(s)
I20260620 07:07:01.269832 27274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:07:01.278612 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27274
I20260620 07:07:01.278813 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:07:01.279366 11591 external_mini_cluster.cc:1428] Setting key a3cc6a59ae4261ff85a44937ad4bc257
Jun 20 07:07:01 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939221, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:01.292506 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.272515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:58237 (local address 127.11.81.254:35801)
0620 07:07:01.272722 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.272728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.274035 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 07:07:01.276058 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.276064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.276167 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.276258 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:01.277940 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.278892 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.281025 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.281531 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.281686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.286043 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:01.286049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:01.286053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:01.286092 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:01.288732 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.289703 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.289710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.289714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.289937 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.290364 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.290369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.290372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.290768 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:01.290987 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.291924 (+ 937us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.292173 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:07:01.293443 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.273066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:58237)
0620 07:07:01.273839 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:01.273884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:01.275732 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:01.276552 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:01.276562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:01.277121 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:01.277758 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.277779 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.279080 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.279086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.280858 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.280876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.281778 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.281786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.282085 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.282829 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:01.282859 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:01.285668 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:01.288897 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:01.288908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:01.288928 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:01.289525 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:01.290047 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:01.290051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:01.290053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:01.290241 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:01.290963 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:01.290971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:01.291471 (+ 500us) client_negotiation.cc:769] Sending connection context
0620 07:07:01.291788 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 07:07:01.292236 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":306,"threads_started":1}
I20260620 07:07:01.294942 27402 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:01.295325 27402 heartbeater.cc:464] Registering TS with master...
I20260620 07:07:01.296237 27402 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
I20260620 07:07:01.298071 27079 ts_manager.cc:194] Registered new tserver with Master: fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129)
I20260620 07:07:01.299008 27079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:58237
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:07:01.334928 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:01.665782 27407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:01.666079 27407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:01.666209 27407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:01.673835 27407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:01.673974 27407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:01.674146 27407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:07:01.684448 27407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:01.685776 27407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:01.687129 27407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:01.700582 27412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:01.700620 27413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:01.700642 27415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:01.701989 27407 server_base.cc:1034] running on GCE node
I20260620 07:07:01.702591 27407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:01.703588 27407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:01.704797 27407 hybrid_clock.cc:648] HybridClock initialized: now 1781939221704717 us; error 75 us; skew 500 ppm
I20260620 07:07:01.705130 27407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:01 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939221, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:01.708411 27407 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:07:01.709968 27407 webserver.cc:460] Webserver started at http://127.11.81.195:45683/ using document root <none> and password file <none>
I20260620 07:07:01.710698 27407 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:01.710819 27407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:01.711117 27407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:07:01.714291 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "b4a37d78f82548918ea4d678b18cf1ff"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "52036280942ba837f615267ade59dea5"
server_key_iv: "b356045ef0f05ee780b44bb38c2f28aa"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.715070 27407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "b4a37d78f82548918ea4d678b18cf1ff"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "52036280942ba837f615267ade59dea5"
server_key_iv: "b356045ef0f05ee780b44bb38c2f28aa"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.720352 27407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:07:01.723881 27422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.725183 27407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:07:01.725339 27407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b4a37d78f82548918ea4d678b18cf1ff"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "52036280942ba837f615267ade59dea5"
server_key_iv: "b356045ef0f05ee780b44bb38c2f28aa"
server_key_version: "encryptionkey@0"
I20260620 07:07:01.725533 27407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:01.740512 27407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:01.744791 27407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:01.745081 27407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:01.746060 27407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:01.747491 27407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:01.747570 27407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.747689 27407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:01.747740 27407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:01.784080 27407 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36843
I20260620 07:07:01.784126 27534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36843 every 8 connection(s)
I20260620 07:07:01.785404 27407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:07:01.789691 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27407
I20260620 07:07:01.789906 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:07:01.790558 11591 external_mini_cluster.cc:1428] Setting key 782948aabe01821ddc3f0c50f473f48f
I20260620 07:07:01.791419 27268 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
Jun 20 07:07:01 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939221, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:01.806860 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.788084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:42949 (local address 127.11.81.254:35801)
0620 07:07:01.788309 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.788314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.789453 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 07:07:01.792052 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.792056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.792168 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.792250 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:01.794306 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.795227 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.796781 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.797225 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.797338 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.800964 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:01.800970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:01.800973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:01.801010 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:01.803649 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.804436 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.804440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.804443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.804586 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.805047 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.805053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.805055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.805358 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:01.805497 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.806280 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.806509 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:07:01.807943 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.788517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:42949)
0620 07:07:01.789243 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:01.789304 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:01.791743 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:01.792558 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:01.792570 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:01.793362 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:01.794125 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.794145 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.795381 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.795387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.796637 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.796654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.797456 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.797463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.797697 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.798426 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:01.798451 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:01.800716 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:01.803813 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:01.803824 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:01.803842 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:01.804296 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:01.804696 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:01.804701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:01.804705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:01.804923 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:01.805488 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:01.805497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:01.805876 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 07:07:01.806133 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 07:07:01.806686 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":265,"threads_started":1}
I20260620 07:07:01.809522 27535 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:01.809875 27535 heartbeater.cc:464] Registering TS with master...
I20260620 07:07:01.810648 27535 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
I20260620 07:07:01.812209 27078 ts_manager.cc:194] Registered new tserver with Master: b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
I20260620 07:07:01.813095 27078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:42949
I20260620 07:07:01.820055 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:07:01.837774 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.824844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.11.81.254:35801)
0620 07:07:01.825075 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.825081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.825329 (+ 248us) server_negotiation.cc:407] Connection header received
0620 07:07:01.825539 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.825544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.825613 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.825699 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:01.827315 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.828084 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.829533 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.829962 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.830094 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.832487 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:01.832492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:01.832495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:01.832532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:01.834981 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.835718 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.835723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.835725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.835901 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:01.836314 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:01.836318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:01.836320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:01.836677 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:01.836930 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.837236 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.837460 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 07:07:01.841929 27078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:07:01.848881 27078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:07:01.872674 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.861496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36843 (local address 127.0.0.1:44680)
0620 07:07:01.862414 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:01.862461 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:01.862657 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:01.864311 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:01.864318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:01.864346 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:01.864801 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.864812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.867177 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.867182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.869738 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.869753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.871677 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.871683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.871895 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.871954 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 07:07:01.872081 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 07:07:01.872281 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":755,"threads_started":1}
I20260620 07:07:01.874135 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.861965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.11.81.195:36843)
0620 07:07:01.863666 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.863675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.863709 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:07:01.863811 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.863819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.864091 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.865328 (+ 1237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:01.865617 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.867028 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.869968 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.871459 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.873334 (+ 1875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.873580 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.873653 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.873820 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1907,"threads_started":1}
I20260620 07:07:01.874177 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.862217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.11.81.194:43129)
0620 07:07:01.863310 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.863320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.863812 (+ 492us) server_negotiation.cc:407] Connection header received
0620 07:07:01.864141 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.864148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.864373 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.864533 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:01.865463 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.867684 (+ 2221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.871272 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.872725 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.872834 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.873062 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.873661 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.873843 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":797,"threads_started":1}
I20260620 07:07:01.874650 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.861755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:43129 (local address 127.0.0.1:60824)
0620 07:07:01.863707 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:01.863723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:01.863921 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:01.864813 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:01.864817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:01.864844 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:01.865258 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.865271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.867864 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.867881 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.871090 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.871106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.873286 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.873293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.873467 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.873498 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:07:01.873596 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:07:01.873733 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"thread_start_us":1702,"threads_started":1}
I20260620 07:07:01.878029 27470 tablet_service.cc:1467] Processing CreateTablet for tablet a78438ded61c4d3480e4dfc20e102709 (DEFAULT_TABLE table=test-table [id=b2ea50e90c6547ac8bca2df1480e4080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:01.878274 27337 tablet_service.cc:1467] Processing CreateTablet for tablet a78438ded61c4d3480e4dfc20e102709 (DEFAULT_TABLE table=test-table [id=b2ea50e90c6547ac8bca2df1480e4080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:01.879669 27337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78438ded61c4d3480e4dfc20e102709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:01.879668 27470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78438ded61c4d3480e4dfc20e102709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:01.880718 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.863131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:50010)
0620 07:07:01.872929 (+ 9798us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:01.872948 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:01.873048 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:01.874101 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:01.874106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:01.874133 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:01.874295 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.874307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.876378 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.876383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.878147 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:01.878164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.879947 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.879954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:01.880121 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.880157 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:07:01.880224 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:07:01.880371 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9700,"thread_start_us":9901,"threads_started":1}
I20260620 07:07:01.881882 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:01.863711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.11.81.193:46229)
0620 07:07:01.872962 (+ 9251us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:01.872974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:01.873006 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:07:01.873566 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:01.873576 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:01.873880 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:01.874500 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:01.874741 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.876207 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.878339 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:01.879759 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:01.881083 (+ 1324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:01.881310 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:01.881397 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 07:07:01.881561 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9084,"thread_start_us":8911,"threads_started":1}
I20260620 07:07:01.885985 27203 tablet_service.cc:1467] Processing CreateTablet for tablet a78438ded61c4d3480e4dfc20e102709 (DEFAULT_TABLE table=test-table [id=b2ea50e90c6547ac8bca2df1480e4080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:01.888298 27203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a78438ded61c4d3480e4dfc20e102709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:01.892436 27553 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Bootstrap starting.
I20260620 07:07:01.894507 27554 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Bootstrap starting.
I20260620 07:07:01.897775 27553 tablet_bootstrap.cc:654] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:01.898541 27554 tablet_bootstrap.cc:654] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:01.899164 27553 log.cc:824] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:01.899967 27554 log.cc:824] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:01.902237 27557 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Bootstrap starting.
I20260620 07:07:01.902767 27553 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: No bootstrap required, opened a new log
I20260620 07:07:01.903043 27553 ts_tablet_manager.cc:1397] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 07:07:01.905504 27554 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: No bootstrap required, opened a new log
I20260620 07:07:01.905803 27554 ts_tablet_manager.cc:1397] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 07:07:01.906087 27557 tablet_bootstrap.cc:654] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:01.907068 27557 log.cc:824] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:01.909901 27557 tablet_bootstrap.cc:492] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: No bootstrap required, opened a new log
I20260620 07:07:01.910157 27557 ts_tablet_manager.cc:1397] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:07:01.914714 27553 raft_consensus.cc:348] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.915858 27553 raft_consensus.cc:374] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:01.916015 27553 raft_consensus.cc:729] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd23c0c8c31547b9854265d01255f23b, State: Initialized, Role: FOLLOWER
I20260620 07:07:01.916179 27554 raft_consensus.cc:348] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.916615 27554 raft_consensus.cc:374] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:01.916712 27554 raft_consensus.cc:729] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a37d78f82548918ea4d678b18cf1ff, State: Initialized, Role: FOLLOWER
I20260620 07:07:01.916782 27553 consensus_queue.cc:260] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.917312 27554 consensus_queue.cc:260] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.918105 27402 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
I20260620 07:07:01.917918 27557 raft_consensus.cc:348] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.918259 27557 raft_consensus.cc:374] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:01.918356 27557 raft_consensus.cc:729] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, State: Initialized, Role: FOLLOWER
I20260620 07:07:01.918853 27553 ts_tablet_manager.cc:1428] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260620 07:07:01.918866 27557 consensus_queue.cc:260] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:01.920609 27557 ts_tablet_manager.cc:1428] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 07:07:01.924014 27535 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
I20260620 07:07:01.924458 27554 ts_tablet_manager.cc:1428] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260620 07:07:02.008296 27269 tablet.cc:2369] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:07:02.021812 27403 tablet.cc:2369] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:07:02.037020 27536 tablet.cc:2369] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:07:02.144922 27561 raft_consensus.cc:482] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:02.145251 27561 raft_consensus.cc:504] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:02.147459 27561 leader_election.cc:288] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843), fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129)
I20260620 07:07:02.156491 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.148003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:39529 (local address 127.11.81.194:43129)
0620 07:07:02.148191 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.148197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.149466 (+ 1269us) server_negotiation.cc:407] Connection header received
0620 07:07:02.149661 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.149665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.149738 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.149812 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:02.150332 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.151051 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.153985 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.155245 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.155450 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.155529 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.156077 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.156237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:07:02.156571 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.147841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:43129 (local address 127.11.81.193:39529)
0620 07:07:02.149346 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:02.149361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:02.149519 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:02.149854 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:02.149858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:02.149882 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:02.150175 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:02.150183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.151193 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.151198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:02.153805 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:02.153826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.155685 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.155692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:02.155888 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.155924 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:07:02.156015 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:07:02.156156 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":108,"spinlock_wait_cycles":26496,"thread_start_us":711,"threads_started":1}
I20260620 07:07:02.157603 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.148331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:48933 (local address 127.11.81.195:36843)
0620 07:07:02.148534 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.148540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.148752 (+ 212us) server_negotiation.cc:407] Connection header received
0620 07:07:02.149046 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.149052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.149127 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.149185 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:02.149962 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.151112 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.155442 (+ 4330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.156403 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.156561 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.156628 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.157199 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.157335 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:07:02.157680 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.147775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36843 (local address 127.11.81.193:48933)
0620 07:07:02.148635 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:02.148658 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:02.148883 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:02.149378 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:02.149383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:02.149410 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:02.149771 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:02.149785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.151242 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.151246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:02.153826 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:02.153836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.156825 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.156832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:02.156993 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.157028 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:07:02.157149 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:07:02.157295 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":151,"thread_start_us":1247,"threads_started":1}
I20260620 07:07:02.157874 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78438ded61c4d3480e4dfc20e102709" candidate_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd23c0c8c31547b9854265d01255f23b" is_pre_election: true
I20260620 07:07:02.158416 27357 raft_consensus.cc:2454] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1f96fa7e2c4f80b30d6cdd3ffc16ad in term 0.
I20260620 07:07:02.158638 27490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78438ded61c4d3480e4dfc20e102709" candidate_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a37d78f82548918ea4d678b18cf1ff" is_pre_election: true
I20260620 07:07:02.159163 27490 raft_consensus.cc:2454] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1f96fa7e2c4f80b30d6cdd3ffc16ad in term 0.
I20260620 07:07:02.159159 27156 leader_election.cc:302] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, fd23c0c8c31547b9854265d01255f23b; no voters:
I20260620 07:07:02.159649 27561 raft_consensus.cc:2781] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:07:02.159792 27561 raft_consensus.cc:482] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:07:02.159889 27561 raft_consensus.cc:3037] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:02.163010 27561 raft_consensus.cc:504] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:02.163821 27561 leader_election.cc:288] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 1 election: Requested vote from peers b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843), fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129)
I20260620 07:07:02.164247 27357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78438ded61c4d3480e4dfc20e102709" candidate_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd23c0c8c31547b9854265d01255f23b"
I20260620 07:07:02.164279 27490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a78438ded61c4d3480e4dfc20e102709" candidate_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a37d78f82548918ea4d678b18cf1ff"
I20260620 07:07:02.164443 27357 raft_consensus.cc:3037] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:02.164480 27490 raft_consensus.cc:3037] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:02.166613 27490 raft_consensus.cc:2454] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1f96fa7e2c4f80b30d6cdd3ffc16ad in term 1.
I20260620 07:07:02.166623 27357 raft_consensus.cc:2454] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1f96fa7e2c4f80b30d6cdd3ffc16ad in term 1.
I20260620 07:07:02.167039 27157 leader_election.cc:302] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, b4a37d78f82548918ea4d678b18cf1ff; no voters:
I20260620 07:07:02.167357 27561 raft_consensus.cc:2781] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:07:02.167769 27561 raft_consensus.cc:686] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 LEADER]: Becoming Leader. State: Replica: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, State: Running, Role: LEADER
I20260620 07:07:02.168220 27561 consensus_queue.cc:237] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } }
I20260620 07:07:02.172880 27079 catalog_manager.cc:5495] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193). New cstate: current_term: 1 leader_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } health_report { overall_health: HEALTHY } } }
I20260620 07:07:02.244194 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.238667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.11.81.193:46229)
0620 07:07:02.238840 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.238845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.238863 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:07:02.239001 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.239005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.239085 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.239209 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:07:02.239591 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.240316 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.241968 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.242397 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.242499 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.243012 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:07:02.243622 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:07:02.243736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.243900 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.243970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:07:02.253125 27357 raft_consensus.cc:1261] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Refusing update from remote peer 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:07:02.253126 27490 raft_consensus.cc:1261] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Refusing update from remote peer 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:07:02.254261 27564 consensus_queue.cc:1035] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:07:02.254666 27561 consensus_queue.cc:1035] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:07:02.275460 27569 mvcc.cc:205] Tried to move back new op lower bound from 7298823054336864256 to 7298823054005248000. Current Snapshot: MvccSnapshot[applied={T|T < 7298823054336864256}]
I20260620 07:07:02.282979 27078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:07:02.283372 27078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:07:02.287902 27078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Sending DeleteTablet for 3 replicas of tablet a78438ded61c4d3480e4dfc20e102709
I20260620 07:07:02.291131 27337 tablet_service.cc:1514] Processing DeleteTablet for tablet a78438ded61c4d3480e4dfc20e102709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:02 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:60824
I20260620 07:07:02.291630 27470 tablet_service.cc:1514] Processing DeleteTablet for tablet a78438ded61c4d3480e4dfc20e102709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:02 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:44680
I20260620 07:07:02.292356 27203 tablet_service.cc:1514] Processing DeleteTablet for tablet a78438ded61c4d3480e4dfc20e102709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:02 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 07:07:02.293367 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27140
I20260620 07:07:02.296443 27578 tablet_replica.cc:330] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: stopping tablet replica
I20260620 07:07:02.296978 27579 tablet_replica.cc:330] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: stopping tablet replica
I20260620 07:07:02.297969 27578 raft_consensus.cc:2229] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:07:02.299062 27578 raft_consensus.cc:2258] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:02.301469 27578 ts_tablet_manager.cc:1905] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:07:02.304008 27577 tablet_replica.cc:330] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: stopping tablet replica
I20260620 07:07:02.304514 27577 raft_consensus.cc:2229] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:07:02.304867 27579 raft_consensus.cc:2229] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:07:02.305150 27577 raft_consensus.cc:2258] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:02.306444 27579 raft_consensus.cc:2258] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:02.309569 27579 ts_tablet_manager.cc:1905] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:07:02.310266 27578 ts_tablet_manager.cc:1918] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:07:02.310541 27578 log.cc:1196] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/a78438ded61c4d3480e4dfc20e102709
I20260620 07:07:02.311156 27578 ts_tablet_manager.cc:1939] T a78438ded61c4d3480e4dfc20e102709 P b4a37d78f82548918ea4d678b18cf1ff: Deleting consensus metadata
I20260620 07:07:02.313314 27064 catalog_manager.cc:4841] TS b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): tablet a78438ded61c4d3480e4dfc20e102709 (table test-table [id=b2ea50e90c6547ac8bca2df1480e4080]) successfully deleted
I20260620 07:07:02.315238 27577 ts_tablet_manager.cc:1905] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:07:02 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:07:02.317584 27579 ts_tablet_manager.cc:1918] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:07:02.317864 27579 log.cc:1196] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/a78438ded61c4d3480e4dfc20e102709
I20260620 07:07:02.318557 27579 ts_tablet_manager.cc:1939] T a78438ded61c4d3480e4dfc20e102709 P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Deleting consensus metadata
I20260620 07:07:02.320623 27064 catalog_manager.cc:4841] TS 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229): tablet a78438ded61c4d3480e4dfc20e102709 (table test-table [id=b2ea50e90c6547ac8bca2df1480e4080]) successfully deleted
I20260620 07:07:02.322474 27577 ts_tablet_manager.cc:1918] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:07:02.322656 27577 log.cc:1196] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/a78438ded61c4d3480e4dfc20e102709
I20260620 07:07:02.323199 27577 ts_tablet_manager.cc:1939] T a78438ded61c4d3480e4dfc20e102709 P fd23c0c8c31547b9854265d01255f23b: Deleting consensus metadata
I20260620 07:07:02.324733 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.294486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.11.81.193:46229)
0620 07:07:02.294646 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.294652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.304346 (+ 9694us) server_negotiation.cc:407] Connection header received
0620 07:07:02.304601 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.304607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.304711 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.304832 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:02.307091 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.308234 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.310506 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.311233 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.311383 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.317147 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:02.317163 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:02.317181 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:02.317236 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:02.321420 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.322197 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.322204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.322207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.322364 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.322890 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.322896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.322900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.323243 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:02.323598 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.324250 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.324479 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:07:02.325297 27063 catalog_manager.cc:4841] TS fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): tablet a78438ded61c4d3480e4dfc20e102709 (table test-table [id=b2ea50e90c6547ac8bca2df1480e4080]) successfully deleted
I20260620 07:07:02.326063 27263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:02.422899 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27140
I20260620 07:07:02.438205 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27274
Jun 20 07:07:02 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:07:02.453320 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.438931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.11.81.194:43129)
0620 07:07:02.439126 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.439133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.439309 (+ 176us) server_negotiation.cc:407] Connection header received
0620 07:07:02.439486 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.439490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.439551 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.439633 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:02.441035 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.441735 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.443004 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.443474 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.443584 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.447847 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:02.447863 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:02.447879 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:02.447919 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:02.450398 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.451143 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.451150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.451153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.451355 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.451736 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.451741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.451745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.452144 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:02.452439 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.452744 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.452964 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:07:02.453953 27397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:02.543251 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27274
I20260620 07:07:02.557082 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27407
Jun 20 07:07:02 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:07:02.572486 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:02.557818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.11.81.195:36843)
0620 07:07:02.557978 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:02.557984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:02.558019 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:07:02.558247 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:02.558251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:02.558325 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:02.558428 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:02.559911 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.560648 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.561926 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:02.562320 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:02.562421 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:02.566742 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:02.566760 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:02.566813 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:02.566854 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:02.569590 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.570325 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.570332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.570334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.570490 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:02.570873 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:02.570878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:02.570880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:02.571268 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:02.571733 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:02.572058 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 07:07:02.572230 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:07:02.573130 27530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:02.661032 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27407
I20260620 07:07:02.674590 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27047
I20260620 07:07:02.675832 27109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:02.784092 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27047
I20260620 07:07:22.802348 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--webserver_interface=127.11.81.254
--webserver_port=34871
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:23.138566 27585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:23.138882 27585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:23.139021 27585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:23.146522 27585 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:07:23.146642 27585 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:07:23.146699 27585 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:07:23.146767 27585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:07:23.146843 27585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:23.146944 27585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:23.147011 27585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:07:23.147085 27585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:07:23.157577 27585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=34871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:23.158922 27585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:23.160338 27585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:23.182587 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:23.182606 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:23.182636 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:23.183030 27585 server_base.cc:1034] running on GCE node
I20260620 07:07:23.184170 27585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:23.185902 27585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:23.187145 27585 hybrid_clock.cc:648] HybridClock initialized: now 1781939243187073 us; error 50 us; skew 500 ppm
I20260620 07:07:23.187577 27585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:23 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939243, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:23.191702 27585 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:07:23.193621 27585 webserver.cc:460] Webserver started at http://127.11.81.254:34871/ using document root <none> and password file <none>
I20260620 07:07:23.194506 27585 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:23.194636 27585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:23.201561 27585 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:07:23.205186 27601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:23.206774 27585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:07:23.206954 27585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c0fac99e8ccd4513a990a0ac71e553d4"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "f6543201f9125fd1964d3046319b0529"
server_key_iv: "002037588ea6ba90e3fc9a32a3554056"
server_key_version: "encryptionkey@0"
I20260620 07:07:23.208169 27585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:23.222232 27585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:23.226991 27585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:23.227264 27585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:23.245142 27585 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35801
I20260620 07:07:23.245184 27652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35801 every 8 connection(s)
I20260620 07:07:23.246826 27585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:07:23.254356 27653 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:07:23.256247 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27585
I20260620 07:07:23.257097 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:46229
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=38897
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:07:23.258033 27653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap starting.
I20260620 07:07:23.283758 27653 log.cc:824] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:23.300736 27653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:07:23.301566 27653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap complete.
I20260620 07:07:23.315392 27653 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:23.316648 27653 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Initialized, Role: FOLLOWER
I20260620 07:07:23.317430 27653 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:23.317783 27653 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:07:23.317991 27653 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:07:23.318187 27653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:07:23.323966 27653 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:23.324733 27653 leader_election.cc:302] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fac99e8ccd4513a990a0ac71e553d4; no voters:
I20260620 07:07:23.325474 27653 leader_election.cc:288] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:07:23.325615 27657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:07:23.326802 27657 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 LEADER]: Becoming Leader. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Running, Role: LEADER
I20260620 07:07:23.327450 27657 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:23.328099 27653 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:07:23.330027 27659 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fac99e8ccd4513a990a0ac71e553d4. Latest consensus state: current_term: 2 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:23.330022 27658 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:23.330444 27658 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:23.330448 27659 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:23.344816 27662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:07:23.353564 27662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:07:23.355185 27662 catalog_manager.cc:1259] Loaded cluster ID: 82f9bcef75c5409c92788bd629126eb7
I20260620 07:07:23.355406 27662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:07:23.360101 27662 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:07:23.362035 27662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Loaded TSK: 0
I20260620 07:07:23.399381 27662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Generated new TSK 1
I20260620 07:07:23.399825 27662 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:07:23.638072 27655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:23.638370 27655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:23.638497 27655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:23.646019 27655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:23.646145 27655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:23.646296 27655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:07:23.657145 27655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:46229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=38897
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:23.658499 27655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:23.660076 27655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:23.675393 27681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:23.675469 27680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:23.676651 27683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:23.676986 27655 server_base.cc:1034] running on GCE node
I20260620 07:07:23.677623 27655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:23.678673 27655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:23.679903 27655 hybrid_clock.cc:648] HybridClock initialized: now 1781939243679822 us; error 66 us; skew 500 ppm
I20260620 07:07:23.680248 27655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:23 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939243, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:23.689145 27655 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:07:23.690825 27655 webserver.cc:460] Webserver started at http://127.11.81.193:38897/ using document root <none> and password file <none>
I20260620 07:07:23.691663 27655 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:23.691814 27655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:23.698105 27655 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:07:23.701246 27690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:23.702766 27655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:07:23.702930 27655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "6e3afec0722af5181afb6c406413c5a9"
server_key_iv: "f965aae0c3f7c0b13b3c7b1b950b91b7"
server_key_version: "encryptionkey@0"
I20260620 07:07:23.704196 27655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:23.724877 27655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:23.729398 27655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:23.729665 27655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:23.730634 27655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:23.732106 27655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:23.732194 27655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:23.732299 27655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:23.732366 27655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:23.763471 27655 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:46229
I20260620 07:07:23.763530 27802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:46229 every 8 connection(s)
I20260620 07:07:23.764987 27655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:07:23.766507 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27655
I20260620 07:07:23.767234 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:43129
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=45621
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:07:23 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939243, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:23.789785 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:23.768211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:52121 (local address 127.11.81.254:35801)
0620 07:07:23.768898 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:23.768914 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:23.769431 (+ 517us) server_negotiation.cc:407] Connection header received
0620 07:07:23.772085 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:23.772099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:23.772462 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:23.772763 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:23.774609 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:23.775908 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:23.777677 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:23.778414 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:23.778595 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:23.782260 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:23.782268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:23.782283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:23.782322 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:23.785158 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:23.785920 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:23.785927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:23.785933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:23.786115 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:23.786544 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:23.786548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:23.786550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:23.787055 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:23.787236 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:23.788086 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 07:07:23.788495 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260620 07:07:23.789788 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:23.768366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:52121)
0620 07:07:23.769216 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:23.769292 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:23.771498 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:23.772965 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:23.772976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:23.773681 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:23.774408 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:23.774433 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:23.776101 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:23.776106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:23.777539 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:23.777553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:23.778662 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:23.778670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:23.778939 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:23.779634 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:23.779660 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:23.781966 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:23.785312 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:23.785322 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:23.785338 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:23.785775 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:23.786231 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:23.786236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:23.786240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:23.786426 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:23.787190 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:23.787198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:23.787583 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 07:07:23.787843 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 07:07:23.788416 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":326,"threads_started":1}
I20260620 07:07:23.792735 27803 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:23.793107 27803 heartbeater.cc:464] Registering TS with master...
I20260620 07:07:23.794018 27803 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
I20260620 07:07:23.796619 27618 ts_manager.cc:194] Registered new tserver with Master: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229)
I20260620 07:07:23.798156 27618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:52121
W20260620 07:07:24.096940 27807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:24.097229 27807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:24.097352 27807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:24.104643 27807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:24.104765 27807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:24.104908 27807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:07:24.115432 27807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:43129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=45621
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:24.116755 27807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:24.118146 27807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:24.132254 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:24.132212 27813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:24.132174 27815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:24.133174 27807 server_base.cc:1034] running on GCE node
I20260620 07:07:24.133839 27807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:24.134886 27807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:24.136104 27807 hybrid_clock.cc:648] HybridClock initialized: now 1781939244136046 us; error 48 us; skew 500 ppm
I20260620 07:07:24.136456 27807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:24 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939244, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:24.139755 27807 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:07:24.141304 27807 webserver.cc:460] Webserver started at http://127.11.81.194:45621/ using document root <none> and password file <none>
I20260620 07:07:24.142000 27807 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:24.142131 27807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:24.147931 27807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:07:24.150943 27822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.152561 27807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:24.152765 27807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "fd23c0c8c31547b9854265d01255f23b"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "89e6407384684bd5af8e631d8761e87d"
server_key_iv: "bfc67097f4835690ef22d57c578a2f43"
server_key_version: "encryptionkey@0"
I20260620 07:07:24.153815 27807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:24.170961 27807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:24.175472 27807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:24.175714 27807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:24.176699 27807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:24.178095 27807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:24.178184 27807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.178267 27807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:24.178341 27807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.209542 27807 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:43129
I20260620 07:07:24.209606 27934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:43129 every 8 connection(s)
I20260620 07:07:24.211006 27807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:07:24.212782 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27807
I20260620 07:07:24.213392 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:36843
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=45683
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:07:24 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939244, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:24.237241 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.217028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60715 (local address 127.11.81.254:35801)
0620 07:07:24.217242 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.217250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.218701 (+ 1451us) server_negotiation.cc:407] Connection header received
0620 07:07:24.220707 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.220712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.220798 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.220886 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:24.223181 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.223983 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.226192 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.226672 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.226780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.231014 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:24.231020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:24.231023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:24.231070 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:24.233769 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.234686 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.234690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.234692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.234812 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.235342 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.235347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.235350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.235640 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:24.235752 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.236727 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.236956 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:07:24.238715 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.217601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:60715)
0620 07:07:24.218512 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.218568 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.220401 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.221189 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.221201 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.221913 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:24.222940 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.222966 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.224176 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.224182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.226010 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.226024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.226900 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.226907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.227209 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.227957 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:24.227993 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:24.230725 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:24.233925 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:24.233935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:24.233955 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:24.234574 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:24.234938 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:24.234943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:24.234946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:24.235216 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:24.235763 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:24.235772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:24.236264 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.236587 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.237186 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":400,"threads_started":1}
I20260620 07:07:24.240288 27935 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:24.240612 27935 heartbeater.cc:464] Registering TS with master...
I20260620 07:07:24.241364 27935 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
I20260620 07:07:24.242990 27618 ts_manager.cc:194] Registered new tserver with Master: fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129)
I20260620 07:07:24.243842 27618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:60715
W20260620 07:07:24.544153 27938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:24.544472 27938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:24.544615 27938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:24.552000 27938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:24.552137 27938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:24.552290 27938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:07:24.562599 27938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:36843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=45683
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:24.563983 27938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:24.565344 27938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:24.578683 27943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:24.578649 27946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:24.578723 27944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:24.584792 27938 server_base.cc:1034] running on GCE node
I20260620 07:07:24.585433 27938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:24.586463 27938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:24.587666 27938 hybrid_clock.cc:648] HybridClock initialized: now 1781939244587603 us; error 60 us; skew 500 ppm
I20260620 07:07:24.587981 27938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:24 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939244, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:24.591176 27938 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:07:24.592708 27938 webserver.cc:460] Webserver started at http://127.11.81.195:45683/ using document root <none> and password file <none>
I20260620 07:07:24.593398 27938 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:24.593531 27938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:24.599203 27938 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:07:24.602156 27953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.603593 27938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:24.603763 27938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b4a37d78f82548918ea4d678b18cf1ff"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "52036280942ba837f615267ade59dea5"
server_key_iv: "b356045ef0f05ee780b44bb38c2f28aa"
server_key_version: "encryptionkey@0"
I20260620 07:07:24.604825 27938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:24.626680 27938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:24.631160 27938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:24.631449 27938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:24.632458 27938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:24.633926 27938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:24.634023 27938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.634135 27938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:24.634209 27938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:24.664572 27938 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36843
I20260620 07:07:24.664569 28065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36843 every 8 connection(s)
I20260620 07:07:24.666045 27938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:07:24.670156 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 27938
Jun 20 07:07:24 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939244, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:07:24.686496 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.671046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.11.81.254:35801)
0620 07:07:24.671538 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.671545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.671920 (+ 375us) server_negotiation.cc:407] Connection header received
0620 07:07:24.672081 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.672086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.672176 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.672273 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:24.673803 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.674631 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.676112 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.676716 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.676810 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.679874 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:24.679881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:24.679886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:24.679929 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:24.682978 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.683961 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.683967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.683971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.684145 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.684643 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.684648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.684652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.685006 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:24.685247 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.685783 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.686000 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":360,"spinlock_wait_cycles":21632,"thread_start_us":256,"threads_started":1}
I20260620 07:07:24.687706 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.668491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33129 (local address 127.11.81.254:35801)
0620 07:07:24.668657 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.668661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.669918 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 07:07:24.672081 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.672086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.672160 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.672236 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:24.673791 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.674593 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.676237 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.676747 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.676862 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.680747 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:24.680754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:24.680758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:24.680801 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:24.683892 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.684870 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.684874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.684876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.685011 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.685512 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.685517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.685520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.685956 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:24.686101 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.687136 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.687370 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":6016}
I20260620 07:07:24.689134 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.668972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:33129)
0620 07:07:24.669738 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.669786 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.671794 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.672517 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.672526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.673056 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:24.673623 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.673640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.674753 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.674759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.676096 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.676110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.676979 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.676987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.677207 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.677880 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:24.677907 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:24.680419 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:24.684053 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:24.684064 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:24.684086 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:24.684734 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:24.685124 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:24.685129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:24.685132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:24.685371 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:24.686120 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:24.686131 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:24.686713 (+ 582us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.687039 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.687633 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":258,"threads_started":1}
I20260620 07:07:24.690800 28066 heartbeater.cc:347] Connected to a master server at 127.11.81.254:35801
I20260620 07:07:24.691108 28066 heartbeater.cc:464] Registering TS with master...
I20260620 07:07:24.691876 28066 heartbeater.cc:511] Master 127.11.81.254:35801 requested a full tablet report, sending...
I20260620 07:07:24.693463 27618 ts_manager.cc:194] Registered new tserver with Master: b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
I20260620 07:07:24.694269 27618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:33129
I20260620 07:07:24.702589 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:07:24.709408 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.703566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.11.81.254:35801)
0620 07:07:24.703734 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.703738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.704204 (+ 466us) server_negotiation.cc:407] Connection header received
0620 07:07:24.704401 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.704404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.704568 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.704699 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:07:24.705146 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.705909 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.707438 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.707853 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.707947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.708521 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:07:24.708739 (+ 218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 07:07:24.709114 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37188: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 07:07:24.709586 27805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37188: token expired
W20260620 07:07:24.710176 27540 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.81.254:35801)
I20260620 07:07:24.723143 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.710666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.11.81.254:35801)
0620 07:07:24.710841 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.710845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.710861 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:07:24.710968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.710972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.711032 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.711119 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:24.712587 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.713424 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.714801 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.715201 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.715334 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.717909 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:24.717914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:24.717917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:24.717953 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:24.720410 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.721194 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.721198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.721201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.721321 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:24.721686 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:24.721691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:24.721694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:24.722033 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:24.722270 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.722596 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.722827 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:07:24.726725 27618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:07:24.734326 27618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:07:24.758881 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.747014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.11.81.193:46229)
0620 07:07:24.747652 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.747661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.748701 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 07:07:24.749243 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.749251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.749538 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.749731 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:24.750806 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.752402 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.755578 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.757152 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.757279 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.757531 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.758416 (+ 885us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.758600 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":352,"threads_started":1}
I20260620 07:07:24.760049 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.746979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39612)
0620 07:07:24.748178 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.748592 (+ 414us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.749019 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.750102 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.750111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.750145 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:24.750592 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.750605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.752595 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.752600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.755354 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.755373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.757768 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.757775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.758072 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.758152 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.759126 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.759384 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":858,"threads_started":1}
I20260620 07:07:24.764907 27738 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e (DEFAULT_TABLE table=test-table [id=c14bc5f38b72400587bd9828a8885729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:24.766753 27738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:24.767333 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.747031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:43129 (local address 127.0.0.1:58514)
0620 07:07:24.748178 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.748270 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.748437 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.759192 (+ 10755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.759196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.759222 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:24.759728 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.759741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.761948 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.761954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.764237 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.764254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.766371 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.766379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.766569 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.766611 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.766691 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.766862 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":1,"thread_start_us":741,"threads_started":1}
I20260620 07:07:24.768205 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.749107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.11.81.195:36843)
0620 07:07:24.758785 (+ 9678us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.758794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.758827 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:07:24.758933 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.758941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.759181 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.759464 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:24.760106 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.761507 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.763869 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.765324 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.765441 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.767638 (+ 2197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.767710 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.767884 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9508,"thread_start_us":9312,"threads_started":1}
I20260620 07:07:24.768479 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.748605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36843 (local address 127.0.0.1:54888)
0620 07:07:24.758341 (+ 9736us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.758362 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.758490 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.759435 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.759453 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.759480 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:24.759878 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.759892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.761664 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.761670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.763661 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.763681 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.766839 (+ 3158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.766847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.767025 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.767058 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.767132 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.767279 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9592,"mutex_wait_us":114,"thread_start_us":9439,"threads_started":1}
I20260620 07:07:24.769263 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.747691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.11.81.194:43129)
0620 07:07:24.757929 (+ 10238us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.757938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.757970 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:07:24.758074 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.758082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.758961 (+ 879us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.759817 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:24.760049 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.761736 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.764465 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.766081 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.768543 (+ 2462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.768736 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.768805 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.768981 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10048,"thread_start_us":9940,"threads_started":1}
I20260620 07:07:24.773022 27870 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e (DEFAULT_TABLE table=test-table [id=c14bc5f38b72400587bd9828a8885729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:24.773022 28001 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e (DEFAULT_TABLE table=test-table [id=c14bc5f38b72400587bd9828a8885729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:07:24.774636 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:24.774636 27870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:07:24.783057 28082 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Bootstrap starting.
I20260620 07:07:24.786676 28082 tablet_bootstrap.cc:654] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:24.787788 28082 log.cc:824] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:24.791206 28084 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Bootstrap starting.
I20260620 07:07:24.791599 28082 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: No bootstrap required, opened a new log
I20260620 07:07:24.791901 28082 ts_tablet_manager.cc:1397] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:07:24.792613 28085 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Bootstrap starting.
I20260620 07:07:24.795470 28084 tablet_bootstrap.cc:654] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:24.796129 28085 tablet_bootstrap.cc:654] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Neither blocks nor log segments found. Creating new log.
I20260620 07:07:24.796712 28084 log.cc:824] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:24.797367 28085 log.cc:824] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:24.800030 28084 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: No bootstrap required, opened a new log
I20260620 07:07:24.800367 28084 ts_tablet_manager.cc:1397] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 07:07:24.801748 28085 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: No bootstrap required, opened a new log
I20260620 07:07:24.802031 28085 ts_tablet_manager.cc:1397] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260620 07:07:24.802104 27803 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
I20260620 07:07:24.802659 28082 raft_consensus.cc:348] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.803170 28082 raft_consensus.cc:374] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:24.803279 28082 raft_consensus.cc:729] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, State: Initialized, Role: FOLLOWER
I20260620 07:07:24.803982 28082 consensus_queue.cc:260] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.805910 28082 ts_tablet_manager.cc:1428] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 07:07:24.810724 28084 raft_consensus.cc:348] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.811422 28084 raft_consensus.cc:374] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:24.811167 28085 raft_consensus.cc:348] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.811549 28084 raft_consensus.cc:729] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd23c0c8c31547b9854265d01255f23b, State: Initialized, Role: FOLLOWER
I20260620 07:07:24.811614 28085 raft_consensus.cc:374] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:07:24.811715 28085 raft_consensus.cc:729] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a37d78f82548918ea4d678b18cf1ff, State: Initialized, Role: FOLLOWER
I20260620 07:07:24.812379 28085 consensus_queue.cc:260] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.812414 28084 consensus_queue.cc:260] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.814082 27935 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
I20260620 07:07:24.814917 28084 ts_tablet_manager.cc:1428] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260620 07:07:24.816450 28066 heartbeater.cc:503] Master 127.11.81.254:35801 was elected leader, sending a full tablet report...
I20260620 07:07:24.817122 28085 ts_tablet_manager.cc:1428] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260620 07:07:24.917701 28067 tablet.cc:2369] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:07:24.925010 28090 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:24.925408 28090 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.927045 28090 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
I20260620 07:07:24.938701 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.927732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:38427 (local address 127.11.81.193:46229)
0620 07:07:24.927942 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.927947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.931968 (+ 4021us) server_negotiation.cc:407] Connection header received
0620 07:07:24.932219 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.932226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.932303 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.932367 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:24.933219 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.934187 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.936294 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.937675 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.937874 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.937964 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.938253 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.938394 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:07:24.938851 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.932303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:43125 (local address 127.11.81.195:36843)
0620 07:07:24.932493 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.932498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.933150 (+ 652us) server_negotiation.cc:407] Connection header received
0620 07:07:24.933281 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.933287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.933376 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.933484 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:07:24.934026 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.935083 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.937022 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.937808 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.937954 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.938027 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:24.938495 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:07:24.938634 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:07:24.938760 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.927634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.11.81.194:38427)
0620 07:07:24.931825 (+ 4191us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.931847 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.932032 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.932604 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.932610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.932637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:24.933053 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.933068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.934318 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.934321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.936119 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.936134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.937900 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.937906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.938089 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.938125 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.938220 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.938363 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4048,"mutex_wait_us":30,"thread_start_us":3845,"threads_started":1}
I20260620 07:07:24.939535 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.932194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36843 (local address 127.11.81.194:43125)
0620 07:07:24.933017 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:24.933038 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:24.933162 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:24.933506 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:24.933511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:24.933535 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:07:24.933879 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.933890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.935225 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.935232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.936874 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:24.936885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.938157 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:24.938163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:24.938332 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:24.938359 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:07:24.939130 (+ 771us) client_negotiation.cc:240] Negotiation successful
0620 07:07:24.939251 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":3,"thread_start_us":415,"threads_started":1}
I20260620 07:07:24.940194 27758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c07cd61ef4fb8a1cc60fba7f6b50e" candidate_uuid: "fd23c0c8c31547b9854265d01255f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" is_pre_election: true
I20260620 07:07:24.940877 27758 raft_consensus.cc:2454] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd23c0c8c31547b9854265d01255f23b in term 0.
I20260620 07:07:24.941555 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c07cd61ef4fb8a1cc60fba7f6b50e" candidate_uuid: "fd23c0c8c31547b9854265d01255f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a37d78f82548918ea4d678b18cf1ff" is_pre_election: true
I20260620 07:07:24.941715 27824 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, fd23c0c8c31547b9854265d01255f23b; no voters:
I20260620 07:07:24.942039 28021 raft_consensus.cc:2454] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd23c0c8c31547b9854265d01255f23b in term 0.
I20260620 07:07:24.942189 28090 raft_consensus.cc:2781] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:07:24.942330 28090 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:07:24.942425 28090 raft_consensus.cc:3037] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:24.944959 28090 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.945562 28090 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [CANDIDATE]: Term 1 election: Requested vote from peers 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
I20260620 07:07:24.946064 28021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c07cd61ef4fb8a1cc60fba7f6b50e" candidate_uuid: "fd23c0c8c31547b9854265d01255f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a37d78f82548918ea4d678b18cf1ff"
I20260620 07:07:24.946157 27758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3c07cd61ef4fb8a1cc60fba7f6b50e" candidate_uuid: "fd23c0c8c31547b9854265d01255f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
I20260620 07:07:24.946257 28021 raft_consensus.cc:3037] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:24.946426 27758 raft_consensus.cc:3037] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:07:24.948640 28021 raft_consensus.cc:2454] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd23c0c8c31547b9854265d01255f23b in term 1.
I20260620 07:07:24.949044 27824 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b; no voters:
I20260620 07:07:24.949327 28090 raft_consensus.cc:2781] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:07:24.950192 27758 raft_consensus.cc:2454] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd23c0c8c31547b9854265d01255f23b in term 1.
I20260620 07:07:24.950700 28090 raft_consensus.cc:686] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 1 LEADER]: Becoming Leader. State: Replica: fd23c0c8c31547b9854265d01255f23b, State: Running, Role: LEADER
I20260620 07:07:24.951308 28090 consensus_queue.cc:237] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:24.955955 27617 catalog_manager.cc:5495] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b reported cstate change: term changed from 0 to 1, leader changed from <none> to fd23c0c8c31547b9854265d01255f23b (127.11.81.194). New cstate: current_term: 1 leader_uuid: "fd23c0c8c31547b9854265d01255f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } health_report { overall_health: UNKNOWN } } }
W20260620 07:07:24.963313 27936 tablet.cc:2369] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:07:25.003468 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:24.998017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.11.81.194:43129)
0620 07:07:24.998228 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:24.998235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:24.998341 (+ 106us) server_negotiation.cc:407] Connection header received
0620 07:07:24.998570 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:24.998576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:24.998661 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:24.998796 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:07:24.999169 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:24.999923 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.001269 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.001682 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.001796 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:25.002312 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:07:25.002832 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:07:25.003010 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:25.003069 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 07:07:25.003161 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:07:25.013235 28021 raft_consensus.cc:1261] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Refusing update from remote peer fd23c0c8c31547b9854265d01255f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:07:25.013528 27758 raft_consensus.cc:1261] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Refusing update from remote peer fd23c0c8c31547b9854265d01255f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:07:25.014281 28090 consensus_queue.cc:1035] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:07:25.014698 28093 consensus_queue.cc:1035] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:07:25.020390 27804 tablet.cc:2369] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:07:25.032210 28096 mvcc.cc:205] Tried to move back new op lower bound from 7298823147562799104 to 7298823147324624896. Current Snapshot: MvccSnapshot[applied={T|T < 7298823147562799104}]
I20260620 07:07:25.054248 27617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:07:25.054594 27617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:07:25.059265 27617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Sending DeleteTablet for 3 replicas of tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:25.060688 27870 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:25 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:58514
I20260620 07:07:25.061761 28105 tablet_replica.cc:330] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: stopping tablet replica
I20260620 07:07:25.061767 27738 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:25 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:39612
I20260620 07:07:25.062449 28105 raft_consensus.cc:2229] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:07:25.062942 28001 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:07:25 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:54888
I20260620 07:07:25.063338 28105 raft_consensus.cc:2258] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:25.064232 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27655
I20260620 07:07:25.065825 28105 ts_tablet_manager.cc:1905] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:07:25.071794 28107 tablet_replica.cc:330] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: stopping tablet replica
I20260620 07:07:25.072276 28107 raft_consensus.cc:2229] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:07:25.072825 28107 raft_consensus.cc:2258] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:25.073987 28105 ts_tablet_manager.cc:1918] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:07:25.074275 28105 log.cc:1196] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:25.076264 28105 ts_tablet_manager.cc:1939] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P fd23c0c8c31547b9854265d01255f23b: Deleting consensus metadata
I20260620 07:07:25.078331 27602 catalog_manager.cc:4841] TS fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e (table test-table [id=c14bc5f38b72400587bd9828a8885729]) successfully deleted
I20260620 07:07:25.080845 28107 ts_tablet_manager.cc:1905] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:07:25.088958 28107 ts_tablet_manager.cc:1918] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:07:25.089943 28106 tablet_replica.cc:330] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: stopping tablet replica
I20260620 07:07:25.090551 28106 raft_consensus.cc:2229] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:07:25.091303 28106 raft_consensus.cc:2258] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:07:25.091249 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.065747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.11.81.193:46229)
0620 07:07:25.065943 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:25.065949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:25.065967 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:07:25.066026 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:25.066032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:25.066112 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:25.066207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:25.068195 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.069419 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.073883 (+ 4464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.074598 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.074726 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:25.081710 (+ 6984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:25.081733 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:25.081754 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:25.081808 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:25.085897 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.086970 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.086981 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.086985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.087184 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.087972 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.087977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.087981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.088497 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:25.088806 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:25.090507 (+ 1701us) server_negotiation.cc:299] Negotiation successful
0620 07:07:25.090789 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:07:25.092100 28107 log.cc:1196] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:25.092145 27798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:25.092749 28107 ts_tablet_manager.cc:1939] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P b4a37d78f82548918ea4d678b18cf1ff: Deleting consensus metadata
I20260620 07:07:25.094774 27603 catalog_manager.cc:4841] TS b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e (table test-table [id=c14bc5f38b72400587bd9828a8885729]) successfully deleted
I20260620 07:07:25.186790 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27655
I20260620 07:07:25.210282 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27807
W20260620 07:07:25.210701 27603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:46229 (error 108)
I20260620 07:07:25.227399 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.226928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39626)
0620 07:07:25.227107 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.227220 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:07:25.227775 27603 catalog_manager.cc:4584] TS 7b1f96fa7e2c4f80b30d6cdd3ffc16ad (127.11.81.193:46229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a3c07cd61ef4fb8a1cc60fba7f6b50e: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
I20260620 07:07:25.232141 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.211959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.11.81.194:43129)
0620 07:07:25.212218 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:25.212223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:25.212244 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:07:25.212323 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:25.212328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:25.212409 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:25.212512 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:25.214587 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.215918 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.217974 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.218712 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.218857 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:25.223056 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:25.223076 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:25.223096 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:25.223154 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:25.227614 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.228773 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.228783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.228787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.228975 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.229624 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.229629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.229633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.230136 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:25.230468 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:25.231503 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 07:07:25.231753 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260620 07:07:25.233246 27930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:25.281545 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.280985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39628)
0620 07:07:25.281238 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.281367 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:07:25.282816 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.282407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39640)
0620 07:07:25.282564 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.282666 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:07:25.343768 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.343093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39644)
0620 07:07:25.343426 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.343552 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 07:07:25.345052 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.344625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39656)
0620 07:07:25.344800 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.344894 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:07:25.348852 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27807
I20260620 07:07:25.365449 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27938
I20260620 07:07:25.379981 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.366310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.11.81.195:36843)
0620 07:07:25.366499 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:25.366505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:25.366525 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:07:25.366694 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:25.366699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:25.366771 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:25.366866 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:25.368292 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.369009 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.370371 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:25.370782 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:25.370899 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:25.373786 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:25.373803 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:25.373820 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:25.373859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:25.376894 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.377741 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.377751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.377753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.377918 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:25.378335 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:25.378341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:25.378344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:25.378729 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:25.379016 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:25.379415 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:07:25.379647 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:07:25.380807 28061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:25.425930 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.425325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39662)
0620 07:07:25.425580 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.425703 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:07:25.427157 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.426701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39664)
0620 07:07:25.426910 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.427004 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 07:07:25.473240 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27938
I20260620 07:07:25.488188 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 27585
I20260620 07:07:25.489488 27648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:07:25.605742 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:25.605080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:46229 (local address 127.0.0.1:39672)
0620 07:07:25.605290 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:25.605420 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:07:25.606002 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 27585
I20260620 07:07:45.627650 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--webserver_interface=127.11.81.254
--webserver_port=34871
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:07:45.964581 28113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:45.964887 28113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:45.965022 28113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:45.972431 28113 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:07:45.972535 28113 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:07:45.972596 28113 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:07:45.972668 28113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:07:45.972738 28113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:45.972846 28113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:45.972903 28113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:07:45.972975 28113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:07:45.983130 28113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=34871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:45.984464 28113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:45.985857 28113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:45.998131 28121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:45.998159 28119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:45.998184 28118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:45.998955 28113 server_base.cc:1034] running on GCE node
I20260620 07:07:45.999825 28113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:46.001338 28113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:46.002592 28113 hybrid_clock.cc:648] HybridClock initialized: now 1781939266002507 us; error 52 us; skew 500 ppm
I20260620 07:07:46.002945 28113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:46 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939266, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:46.011803 28113 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:07:46.013350 28113 webserver.cc:460] Webserver started at http://127.11.81.254:34871/ using document root <none> and password file <none>
I20260620 07:07:46.014086 28113 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:46.014180 28113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:46.020083 28113 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:07:46.023305 28128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:46.024912 28113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:07:46.025103 28113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c0fac99e8ccd4513a990a0ac71e553d4"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "f6543201f9125fd1964d3046319b0529"
server_key_iv: "002037588ea6ba90e3fc9a32a3554056"
server_key_version: "encryptionkey@0"
I20260620 07:07:46.026132 28113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:46.047704 28113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:46.052179 28113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:46.052445 28113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:46.071956 28113 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35801
I20260620 07:07:46.071996 28179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35801 every 8 connection(s)
I20260620 07:07:46.073706 28113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:07:46.081033 28180 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:07:46.081807 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28113
I20260620 07:07:46.082784 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:46229
--local_ip_for_outbound_sockets=127.11.81.193
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=38897
--webserver_interface=127.11.81.193
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:07:46.085891 28180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap starting.
I20260620 07:07:46.107999 28180 log.cc:824] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:46.124575 28180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:07:46.125098 28180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Bootstrap complete.
I20260620 07:07:46.134642 28180 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:46.135728 28180 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Initialized, Role: FOLLOWER
I20260620 07:07:46.136437 28180 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:46.136675 28180 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:07:46.136785 28180 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:07:46.136951 28180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:07:46.140936 28180 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:46.141435 28180 leader_election.cc:302] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fac99e8ccd4513a990a0ac71e553d4; no voters:
I20260620 07:07:46.142108 28180 leader_election.cc:288] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:07:46.142256 28184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:07:46.143512 28184 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [term 3 LEADER]: Becoming Leader. State: Replica: c0fac99e8ccd4513a990a0ac71e553d4, State: Running, Role: LEADER
I20260620 07:07:46.144066 28184 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } }
I20260620 07:07:46.144299 28180 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:07:46.146447 28186 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fac99e8ccd4513a990a0ac71e553d4. Latest consensus state: current_term: 3 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:46.146836 28186 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:46.147235 28185 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fac99e8ccd4513a990a0ac71e553d4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35801 } } }
I20260620 07:07:46.147563 28185 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:07:46.156965 28191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:07:46.166309 28191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:07:46.168169 28191 catalog_manager.cc:1259] Loaded cluster ID: 82f9bcef75c5409c92788bd629126eb7
I20260620 07:07:46.168375 28191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:07:46.172650 28191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:07:46.174630 28191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Loaded TSK: 0
I20260620 07:07:46.174928 28191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0fac99e8ccd4513a990a0ac71e553d4: Loaded TSK: 1
F20260620 07:07:46.176236 28191 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6d3f3ce9ad google::LogMessage::Fail() at ??:0
@ 0x7f6d3f3d188a google::LogMessage::SendToLog() at ??:0
@ 0x7f6d3f3ce4de google::LogMessage::Flush() at ??:0
@ 0x7f6d3f3cfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6d431327d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6d4312173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6d54dfd1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6d54dfc783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6d54e882a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6d54e880c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6d54ed25e9 std::function<>::operator()() at ??:0
@ 0x7f6d54e00cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6d54dfec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6d54e86610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6d54e86439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6d4201a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6d4201d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6d4201d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6d41ff8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6d452d26db start_thread at ??:0
@ 0x7f6d3e26861f clone at ??:0
W20260620 07:07:46.478925 28182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:46.479228 28182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:46.479380 28182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:46.487522 28182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:46.487670 28182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:46.487881 28182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:07:46.498461 28182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:46229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=38897
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:46.499850 28182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:46.501188 28182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:46.514525 28228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:46.514494 28230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:46.514556 28227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:46.515416 28182 server_base.cc:1034] running on GCE node
I20260620 07:07:46.516139 28182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:46.517307 28182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:46.518538 28182 hybrid_clock.cc:648] HybridClock initialized: now 1781939266518479 us; error 57 us; skew 500 ppm
I20260620 07:07:46.518889 28182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:46 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939266, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:46.522322 28182 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:07:46.523975 28182 webserver.cc:460] Webserver started at http://127.11.81.193:38897/ using document root <none> and password file <none>
I20260620 07:07:46.524712 28182 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:46.524842 28182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:46.530675 28182 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:07:46.533886 28237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:46.535445 28182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:46.535607 28182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad"
format_stamp: "Formatted at 2026-06-20 07:07:00 on dist-test-slave-wvz3"
server_key: "6e3afec0722af5181afb6c406413c5a9"
server_key_iv: "f965aae0c3f7c0b13b3c7b1b950b91b7"
server_key_version: "encryptionkey@0"
I20260620 07:07:46.536621 28182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:46.562677 28182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:46.567094 28182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:46.567386 28182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:46.568416 28182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:46.570380 28244 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 07:07:46.576352 28182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:07:46.576507 28182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 07:07:46.576639 28182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:07:46.580009 28182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:07:46.580097 28182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260620 07:07:46.580461 28244 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Bootstrap starting.
I20260620 07:07:46.588932 28244 log.cc:824] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Log is configured to *not* fsync() on all Append() calls
I20260620 07:07:46.604395 28244 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:07:46.605151 28244 tablet_bootstrap.cc:492] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Bootstrap complete.
I20260620 07:07:46.605719 28244 ts_tablet_manager.cc:1397] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.007s
I20260620 07:07:46.615034 28244 raft_consensus.cc:348] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:46.616317 28244 raft_consensus.cc:729] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad, State: Initialized, Role: FOLLOWER
I20260620 07:07:46.616346 28182 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:46229
I20260620 07:07:46.616379 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:46229 every 8 connection(s)
I20260620 07:07:46.617110 28244 consensus_queue.cc:260] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:46.617816 28182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:07:46.618873 28244 ts_tablet_manager.cc:1428] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260620 07:07:46.621076 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28182
I20260620 07:07:46.621861 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:43129
--local_ip_for_outbound_sockets=127.11.81.194
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=45621
--webserver_interface=127.11.81.194
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:07:46.622825 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:46.620665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:50109)
0620 07:07:46.621525 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:46.621830 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":364,"threads_started":1}
W20260620 07:07:46.623553 28353 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:35801 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:35801: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
I20260620 07:07:46.624382 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:46.624055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:57391)
0620 07:07:46.624208 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:46.624281 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:07:46.625238 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:46.624910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:53405)
0620 07:07:46.625054 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:46.625126 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 07:07:46.625454 28353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:07:46.950664 28356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:46.950964 28356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:46.951061 28356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:46.958660 28356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:46.958796 28356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:46.958925 28356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:07:46.968977 28356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:43129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=45621
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:46.970233 28356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:46.971591 28356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:46.984405 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:46.984570 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:46.984356 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:46.985472 28356 server_base.cc:1034] running on GCE node
I20260620 07:07:46.986065 28356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:46.987017 28356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:46.988226 28356 hybrid_clock.cc:648] HybridClock initialized: now 1781939266988147 us; error 68 us; skew 500 ppm
I20260620 07:07:46.988554 28356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:46 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939266, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:46.991909 28356 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:07:46.993580 28356 webserver.cc:460] Webserver started at http://127.11.81.194:45621/ using document root <none> and password file <none>
I20260620 07:07:46.994311 28356 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:46.994432 28356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:47.000236 28356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:07:47.003321 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.005007 28356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:07:47.005205 28356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "fd23c0c8c31547b9854265d01255f23b"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "89e6407384684bd5af8e631d8761e87d"
server_key_iv: "bfc67097f4835690ef22d57c578a2f43"
server_key_version: "encryptionkey@0"
I20260620 07:07:47.006299 28356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:47.032650 28356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:47.036969 28356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:47.037245 28356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:47.038290 28356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:47.039686 28356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:47.039790 28356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.039893 28356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:47.039990 28356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.070348 28356 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:43129
I20260620 07:07:47.070408 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:43129 every 8 connection(s)
I20260620 07:07:47.071866 28356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:07:47.075778 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28356
I20260620 07:07:47.076449 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:36843
--local_ip_for_outbound_sockets=127.11.81.195
--tserver_master_addrs=127.11.81.254:35801
--webserver_port=45683
--webserver_interface=127.11.81.195
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:07:47.076903 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.074893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:38627)
0620 07:07:47.075670 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.075942 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":286,"threads_started":1}
W20260620 07:07:47.077574 28484 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:35801 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:35801: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
I20260620 07:07:47.078446 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.078112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:33267)
0620 07:07:47.078257 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.078329 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:07:47.079253 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.078953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:55079)
0620 07:07:47.079074 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.079148 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:07:47.079538 28484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:07:47.407493 28487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:07:47.407792 28487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:07:47.407895 28487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:07:47.415575 28487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:07:47.415701 28487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:07:47.415834 28487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:07:47.425936 28487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:36843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=45683
--tserver_master_addrs=127.11.81.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:07:47.427208 28487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:07:47.428529 28487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:07:47.441732 28493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:47.441852 28495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:07:47.441854 28492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:07:47.442893 28487 server_base.cc:1034] running on GCE node
I20260620 07:07:47.443626 28487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:07:47.444640 28487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:07:47.445881 28487 hybrid_clock.cc:648] HybridClock initialized: now 1781939267445815 us; error 48 us; skew 500 ppm
I20260620 07:07:47.446204 28487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:07:47 dist-test-slave-wvz3 krb5kdc[27034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939267, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:07:47.449347 28487 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:07:47.450886 28487 webserver.cc:460] Webserver started at http://127.11.81.195:45683/ using document root <none> and password file <none>
I20260620 07:07:47.451660 28487 fs_manager.cc:359] Metadata directory not provided
I20260620 07:07:47.451763 28487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:07:47.457427 28487 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:07:47.460544 28502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.462116 28487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:07:47.462285 28487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "b4a37d78f82548918ea4d678b18cf1ff"
format_stamp: "Formatted at 2026-06-20 07:07:01 on dist-test-slave-wvz3"
server_key: "52036280942ba837f615267ade59dea5"
server_key_iv: "b356045ef0f05ee780b44bb38c2f28aa"
server_key_version: "encryptionkey@0"
I20260620 07:07:47.463500 28487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:07:47.483122 28487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:07:47.488034 28487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:07:47.488276 28487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:07:47.489274 28487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:07:47.490880 28487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:07:47.490971 28487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.491079 28487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:07:47.491156 28487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:07:47.523243 28487 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:36843
I20260620 07:07:47.523353 28614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:36843 every 8 connection(s)
I20260620 07:07:47.524725 28487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:07:47.529696 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.527776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:41387)
0620 07:07:47.528566 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.528804 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":325,"threads_started":1}
W20260620 07:07:47.530496 28615 heartbeater.cc:644] Failed to heartbeat to 127.11.81.254:35801 (0 consecutive failures): Network error: Failed to ping master at 127.11.81.254:35801: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
I20260620 07:07:47.530650 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28487
I20260620 07:07:47.531520 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.531129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:55045)
0620 07:07:47.531279 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.531393 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:07:47.532333 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.532004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:35457)
0620 07:07:47.532142 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.532219 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:07:47.532575 28615 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:07:47.626727 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:47.625978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:39313)
0620 07:07:47.626419 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:47.626513 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":194,"threads_started":1}
I20260620 07:07:48.081025 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.080146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:48621)
0620 07:07:48.080632 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:48.080734 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260620 07:07:48.310215 28621 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:48.310536 28621 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:48.312323 28621 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
Jun 20 07:07:48 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939266, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
Jun 20 07:07:48 dist-test-slave-wvz3 krb5kdc[27034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939266, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:07:48.346578 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.312887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:59409 (local address 127.11.81.194:43129)
0620 07:07:48.325733 (+ 12846us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:48.325751 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:48.326213 (+ 462us) server_negotiation.cc:407] Connection header received
0620 07:07:48.328624 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:48.328633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:48.328966 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:48.329244 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:48.331464 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.332647 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.334422 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.335064 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.335245 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:48.339006 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:48.339015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:48.339031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:48.339094 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:48.342650 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:48.343586 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:48.343592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:48.343598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:48.343751 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:48.344181 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:48.344187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:48.344190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:48.344939 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:48.345137 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:48.345840 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 07:07:48.346178 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12632,"thread_start_us":12451,"threads_started":1}
I20260620 07:07:48.346611 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.313015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:55643 (local address 127.11.81.195:36843)
0620 07:07:48.325268 (+ 12253us) server_negotiation.cc:206] Beginning negotiation
0620 07:07:48.325290 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:07:48.326498 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 07:07:48.328382 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:07:48.328392 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:07:48.328800 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:07:48.329099 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:07:48.331475 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.332648 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.334374 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.335000 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.335180 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:48.339369 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:07:48.339381 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:07:48.339401 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:07:48.339453 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:07:48.342730 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:48.343572 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:48.343580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:48.343586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:48.343748 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:07:48.344185 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:07:48.344188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:07:48.344191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:07:48.344714 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:07:48.344855 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:07:48.345810 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 07:07:48.346217 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12023,"thread_start_us":11853,"threads_started":1}
I20260620 07:07:48.346626 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.312690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:43129 (local address 127.11.81.193:59409)
0620 07:07:48.326012 (+ 13322us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:48.326059 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:48.326669 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:48.329260 (+ 2591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:48.329270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:48.330349 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:48.331253 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:48.331276 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.332832 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.332836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:48.334220 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:48.334233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.335274 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.335307 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:48.335523 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:48.336205 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:48.336228 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:48.338629 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:48.342809 (+ 4180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:48.342819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:48.342835 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:48.343427 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:48.343868 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:48.343872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:48.343874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:48.344061 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:48.345074 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:48.345077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:48.345315 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 07:07:48.345556 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 07:07:48.346154 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13167,"mutex_wait_us":226,"spinlock_wait_cycles":35968,"thread_start_us":13034,"threads_started":1}
I20260620 07:07:48.346714 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.312823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:36843 (local address 127.11.81.193:55643)
0620 07:07:48.326368 (+ 13545us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:48.326384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:07:48.326633 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:07:48.329260 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:07:48.329270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:07:48.330335 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:07:48.331261 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:48.331275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.332843 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.332847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:48.334225 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:07:48.334238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:07:48.335197 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:07:48.335203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:07:48.335488 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:07:48.336250 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:07:48.336267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:07:48.339027 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:07:48.342908 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:48.342912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:48.342914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:48.343428 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:48.343886 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:07:48.343891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:07:48.343895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:07:48.344074 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:07:48.344949 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:07:48.344958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:07:48.345321 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 07:07:48.345557 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 07:07:48.346322 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13418,"mutex_wait_us":361,"spinlock_wait_cycles":6656,"thread_start_us":13155,"threads_started":1}
W20260620 07:07:48.349889 28238 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
W20260620 07:07:48.350167 28239 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:48.350337 28239 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad; no voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b
I20260620 07:07:48.350728 28621 raft_consensus.cc:2726] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:07:48.534269 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.533253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:36187)
0620 07:07:48.533741 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:48.533954 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":229,"threads_started":1}
I20260620 07:07:48.628046 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:48.627637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:35493)
0620 07:07:48.627813 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:48.627902 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:07:49.082652 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:49.081916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:50547)
0620 07:07:49.082377 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:49.082463 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260620 07:07:49.535961 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:49.535218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:42349)
0620 07:07:49.535687 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:49.535769 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":264,"threads_started":1}
I20260620 07:07:49.629969 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:49.629063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:34221)
0620 07:07:49.629575 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:49.629680 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260620 07:07:50.084447 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:50.083672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:49187)
0620 07:07:50.084178 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:50.084268 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":285,"threads_started":1}
I20260620 07:07:50.515475 28631 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:50.515686 28631 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:50.516396 28631 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
W20260620 07:07:50.517480 28239 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
W20260620 07:07:50.517657 28238 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:50.517781 28238 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad; no voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b
I20260620 07:07:50.518044 28631 raft_consensus.cc:2726] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:07:50.537696 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:50.536863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:56299)
0620 07:07:50.537444 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:50.537527 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":292,"threads_started":1}
I20260620 07:07:50.631841 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:50.631027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:52951)
0620 07:07:50.631538 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:50.631649 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":281,"threads_started":1}
I20260620 07:07:51.086153 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:51.085368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:32811)
0620 07:07:51.085855 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:51.085961 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":237,"threads_started":1}
I20260620 07:07:51.539551 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:51.538610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:45349)
0620 07:07:51.539141 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:51.539234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":245,"threads_started":1}
I20260620 07:07:51.633790 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:51.632924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:46349)
0620 07:07:51.633481 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:51.633589 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
I20260620 07:07:52.088068 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:52.087244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:50375)
0620 07:07:52.087781 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:52.087879 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":259,"threads_started":1}
I20260620 07:07:52.541488 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:52.540656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:37819)
0620 07:07:52.541193 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:52.541292 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":299,"threads_started":1}
I20260620 07:07:52.635651 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:52.634805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:47701)
0620 07:07:52.635360 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:52.635477 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260620 07:07:53.089897 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:53.089108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:56391)
0620 07:07:53.089610 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:53.089712 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
I20260620 07:07:53.119880 28641 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:53.120081 28641 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:53.120724 28641 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
W20260620 07:07:53.121843 28238 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
W20260620 07:07:53.122030 28239 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:53.122153 28239 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad; no voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b
I20260620 07:07:53.122390 28641 raft_consensus.cc:2726] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:07:53.543239 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:53.542450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:56933)
0620 07:07:53.542953 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:53.543051 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260620 07:07:53.637550 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:53.636766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:48295)
0620 07:07:53.637230 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:53.637354 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
I20260620 07:07:54.091831 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:54.090939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:40565)
0620 07:07:54.091541 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:54.091644 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":248,"threads_started":1}
I20260620 07:07:54.545027 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:54.544232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:35989)
0620 07:07:54.544757 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:54.544846 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":243,"threads_started":1}
I20260620 07:07:54.639420 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:54.638526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:55161)
0620 07:07:54.639118 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:54.639214 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260620 07:07:55.093770 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:55.092971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:37459)
0620 07:07:55.093465 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:55.093577 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260620 07:07:55.546779 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:55.546018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:43191)
0620 07:07:55.546502 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:55.546592 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260620 07:07:55.641158 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:55.640367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:44241)
0620 07:07:55.640865 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:55.640955 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260620 07:07:56.095611 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:56.094815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:55781)
0620 07:07:56.095326 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:56.095447 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260620 07:07:56.116216 28651 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:56.116444 28651 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:56.117108 28651 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
W20260620 07:07:56.118002 28239 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
W20260620 07:07:56.118180 28238 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:56.118278 28238 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad; no voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b
I20260620 07:07:56.118502 28651 raft_consensus.cc:2726] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:07:56.548636 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:56.547797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:40287)
0620 07:07:56.548367 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:56.548472 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260620 07:07:56.642963 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:56.642240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:60139)
0620 07:07:56.642696 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:56.642789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260620 07:07:57.097602 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:57.096730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:53041)
0620 07:07:57.097281 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:57.097407 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":305,"threads_started":1}
I20260620 07:07:57.550310 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:57.549615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:44331)
0620 07:07:57.550050 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:57.550133 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260620 07:07:57.644773 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:57.643964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:39293)
0620 07:07:57.644484 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:57.644586 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260620 07:07:58.099455 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:58.098618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:33671)
0620 07:07:58.099163 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:58.099253 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260620 07:07:58.552235 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:58.551403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:48225)
0620 07:07:58.551968 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:58.552071 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":249,"threads_started":1}
I20260620 07:07:58.646596 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:58.645783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:48905)
0620 07:07:58.646344 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:58.646437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260620 07:07:59.101176 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:59.100384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:58087)
0620 07:07:59.100908 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:59.100998 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":288,"threads_started":1}
I20260620 07:07:59.534560 28661 raft_consensus.cc:482] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:07:59.534739 28661 raft_consensus.cc:504] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f96fa7e2c4f80b30d6cdd3ffc16ad" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 46229 } } peers { permanent_uuid: "fd23c0c8c31547b9854265d01255f23b" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 43129 } } peers { permanent_uuid: "b4a37d78f82548918ea4d678b18cf1ff" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 36843 } }
I20260620 07:07:59.535475 28661 leader_election.cc:288] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129), b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843)
W20260620 07:07:59.536450 28239 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a37d78f82548918ea4d678b18cf1ff (127.11.81.195:36843): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
W20260620 07:07:59.536631 28238 leader_election.cc:341] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd23c0c8c31547b9854265d01255f23b (127.11.81.194:43129): Not found: Tablet not found: 9a3c07cd61ef4fb8a1cc60fba7f6b50e
I20260620 07:07:59.536749 28238 leader_election.cc:302] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b1f96fa7e2c4f80b30d6cdd3ffc16ad; no voters: b4a37d78f82548918ea4d678b18cf1ff, fd23c0c8c31547b9854265d01255f23b
I20260620 07:07:59.536983 28661 raft_consensus.cc:2726] T 9a3c07cd61ef4fb8a1cc60fba7f6b50e P 7b1f96fa7e2c4f80b30d6cdd3ffc16ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:07:59.553938 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:59.553237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:57055)
0620 07:07:59.553684 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:59.553768 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260620 07:07:59.648530 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:07:59.647702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:56701)
0620 07:07:59.648185 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 07:07:59.648298 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260620 07:08:00.103257 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:00.102325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:42187)
0620 07:08:00.102886 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:00.103007 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":270,"threads_started":1}
I20260620 07:08:00.555914 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:00.555076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:43085)
0620 07:08:00.555645 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:00.555748 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":323,"threads_started":1}
I20260620 07:08:00.650229 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:00.649434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:52505)
0620 07:08:00.649949 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:00.650049 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
I20260620 07:08:01.105197 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:01.104397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:48669)
0620 07:08:01.104917 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:01.105007 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
I20260620 07:08:01.557636 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:01.556862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:53743)
0620 07:08:01.557355 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:01.557439 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
I20260620 07:08:01.652451 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:01.651585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.193:42713)
0620 07:08:01.652153 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:01.652254 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260620 07:08:02.107369 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:02.106386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.194:56653)
0620 07:08:02.107025 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:02.107148 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":278,"threads_started":1}
I20260620 07:08:02.540244 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28182
I20260620 07:08:02.557636 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:02.542517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.11.81.193:46229)
0620 07:08:02.543084 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:02.543091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:02.543472 (+ 381us) server_negotiation.cc:407] Connection header received
0620 07:08:02.543663 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:02.543670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:02.543932 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:02.544074 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:02.545447 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.546441 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.547802 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.548406 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.548621 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:02.551226 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:02.551240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:02.551255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:02.551356 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:02.554527 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.555414 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.555423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.555425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.555585 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.556010 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.556027 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.556029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.556465 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:02.556724 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:02.557055 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 07:08:02.557291 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":359,"threads_started":1}
I20260620 07:08:02.559036 28347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:02.559587 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:02.558745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:35801 (local address 127.11.81.195:50067)
0620 07:08:02.559266 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:02.559430 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.254:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260620 07:08:02.649945 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28182
I20260620 07:08:02.665385 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28356
I20260620 07:08:02.679768 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:02.666189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.11.81.194:43129)
0620 07:08:02.666692 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:02.666697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:02.666714 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:08:02.666769 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:02.666773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:02.666833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:02.666949 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:02.668473 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.669301 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.670627 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.671055 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.671147 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:02.674231 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:02.674237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:02.674241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:02.674280 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:02.677030 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.677779 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.677782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.677785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.677913 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.678258 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.678261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.678264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.678549 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:02.678755 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:02.679087 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 07:08:02.679265 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260620 07:08:02.680562 28479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:02.764981 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28356
I20260620 07:08:02.779243 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28487
I20260620 07:08:02.793077 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:02.780132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.11.81.195:36843)
0620 07:08:02.780671 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:02.780676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:02.780694 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:08:02.780749 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:02.780753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:02.780822 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:02.780917 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:02.782256 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.783056 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.784564 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:02.785009 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:02.785109 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:02.787801 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:02.787806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:02.787810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:02.787848 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:02.790638 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.791319 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.791322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.791325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.791513 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:02.791807 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:02.791810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:02.791812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:02.792072 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:02.792270 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:02.792584 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:08:02.792765 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
I20260620 07:08:02.793753 28610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:02.876695 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28487
2026-06-20T07:08:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64448 ms)
[----------] 30 tests from SecurityITest (290264 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:02 dist-test-slave-wvz3 krb5kdc[28680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:02 dist-test-slave-wvz3 krb5kdc[28680](info): set up 2 sockets
Jun 20 07:08:02 dist-test-slave-wvz3 krb5kdc[28680](info): commencing operation
krb5kdc: starting...
W20260620 07:08:03.899387 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:04 dist-test-slave-wvz3 krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:04.152148 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44883
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:44883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:04.492596 28693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:04.492928 28693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:04.493074 28693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:04.500674 28693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:04.500819 28693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:04.500900 28693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:04.500978 28693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:04.501040 28693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:04.511425 28693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:44883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:04.512851 28693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:04.514206 28693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:04.526355 28698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:04.526733 28699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:04.527357 28701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:04.528838 28693 server_base.cc:1034] running on GCE node
I20260620 07:08:04.529742 28693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:04.531252 28693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:04.532555 28693 hybrid_clock.cc:648] HybridClock initialized: now 1781939284532508 us; error 47 us; skew 500 ppm
I20260620 07:08:04.533023 28693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:04 dist-test-slave-wvz3 krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:04.540436 28693 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:04.542169 28693 webserver.cc:460] Webserver started at http://127.11.81.254:40577/ using document root <none> and password file <none>
I20260620 07:08:04.542964 28693 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:04.543087 28693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:04.543452 28693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:04.546913 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "b9514ece5b8e4aa8bc153355a5cea451"
format_stamp: "Formatted at 2026-06-20 07:08:04 on dist-test-slave-wvz3"
server_key: "a9f11f9d75b0169eb92125fd2623c45f"
server_key_iv: "071d3801a03482529a3605e3a684ca38"
server_key_version: "encryptionkey@0"
I20260620 07:08:04.548003 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "b9514ece5b8e4aa8bc153355a5cea451"
format_stamp: "Formatted at 2026-06-20 07:08:04 on dist-test-slave-wvz3"
server_key: "a9f11f9d75b0169eb92125fd2623c45f"
server_key_iv: "071d3801a03482529a3605e3a684ca38"
server_key_version: "encryptionkey@0"
I20260620 07:08:04.554054 28693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 07:08:04.557962 28708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:04.559578 28693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:04.559768 28693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b9514ece5b8e4aa8bc153355a5cea451"
format_stamp: "Formatted at 2026-06-20 07:08:04 on dist-test-slave-wvz3"
server_key: "a9f11f9d75b0169eb92125fd2623c45f"
server_key_iv: "071d3801a03482529a3605e3a684ca38"
server_key_version: "encryptionkey@0"
I20260620 07:08:04.559981 28693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:04.589174 28693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:04.593644 28693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:04.593946 28693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:04.612182 28693 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:44883
I20260620 07:08:04.612202 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:44883 every 8 connection(s)
I20260620 07:08:04.613615 28693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:04.617033 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28693
I20260620 07:08:04.617226 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:04.617614 28760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:04.618008 11591 external_mini_cluster.cc:1428] Setting key 83db35b75f9a3cb4930b0fd70c09ee75
I20260620 07:08:04.632496 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: Bootstrap starting.
Jun 20 07:08:04 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:04.636351 28760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:04.637552 28760 log.cc:824] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:04.641057 28760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: No bootstrap required, opened a new log
I20260620 07:08:04.642496 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:04.620037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.11.81.254:44883)
0620 07:08:04.620863 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:04.620877 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:04.620926 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:08:04.622624 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:04.622649 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:04.623104 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:04.623517 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:04.625339 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:04.626682 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:04.628567 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:04.629445 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:04.629622 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:04.634911 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:04.634922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:04.634938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:04.634984 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:04.638248 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:04.639040 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:04.639048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:04.639060 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:04.639247 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:04.639697 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:04.639700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:04.639703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:04.640265 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:04.640507 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:04.640973 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 07:08:04.641425 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":380,"threads_started":1}
I20260620 07:08:04.648913 28760 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } }
I20260620 07:08:04.649308 28760 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:04.649412 28760 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9514ece5b8e4aa8bc153355a5cea451, State: Initialized, Role: FOLLOWER
I20260620 07:08:04.649996 28760 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } }
I20260620 07:08:04.650221 28760 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:04.650345 28760 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:04.650504 28760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:04.652864 28760 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } }
I20260620 07:08:04.653486 28760 leader_election.cc:302] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9514ece5b8e4aa8bc153355a5cea451; no voters:
I20260620 07:08:04.654086 28760 leader_election.cc:288] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:04.654220 28765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:04.654834 28765 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [term 1 LEADER]: Becoming Leader. State: Replica: b9514ece5b8e4aa8bc153355a5cea451, State: Running, Role: LEADER
I20260620 07:08:04.655431 28765 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } }
I20260620 07:08:04.656198 28760 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:04.658038 28767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9514ece5b8e4aa8bc153355a5cea451. Latest consensus state: current_term: 1 leader_uuid: "b9514ece5b8e4aa8bc153355a5cea451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } } }
I20260620 07:08:04.658341 28767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:04.659272 28770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:04.659571 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9514ece5b8e4aa8bc153355a5cea451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9514ece5b8e4aa8bc153355a5cea451" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44883 } } }
I20260620 07:08:04.659996 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:04.666425 28770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:04.677811 28770 catalog_manager.cc:1347] Generated new cluster ID: c172d92c2fdd48508336494168908f62
I20260620 07:08:04.677955 28770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:04.694605 28770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:04.696703 28770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:04.707433 28770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: Generated new TSK 0
I20260620 07:08:04.708392 28770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:08:04.777899 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:05.120034 28786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:05.120345 28786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:05.120474 28786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:05.128006 28786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:05.128149 28786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:05.128301 28786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:05.138383 28786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:05.139801 28786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:05.141181 28786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:05.154861 28792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:05.154841 28794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:05.154917 28791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:05.155687 28786 server_base.cc:1034] running on GCE node
I20260620 07:08:05.156354 28786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:05.157429 28786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:05.158715 28786 hybrid_clock.cc:648] HybridClock initialized: now 1781939285158667 us; error 41 us; skew 500 ppm
I20260620 07:08:05.159129 28786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:05 dist-test-slave-wvz3 krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:05.162560 28786 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:08:05.164397 28786 webserver.cc:460] Webserver started at http://127.11.81.193:39727/ using document root <none> and password file <none>
I20260620 07:08:05.165166 28786 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:05.165284 28786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:05.165582 28786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:05.168648 28786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "c1f315ac18ed4eebb3095e10b1ce135b"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "36038111543a24195c27cf593c01fa2f"
server_key_iv: "00253e4cab14157a780bde85891d82cc"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.169533 28786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "c1f315ac18ed4eebb3095e10b1ce135b"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "36038111543a24195c27cf593c01fa2f"
server_key_iv: "00253e4cab14157a780bde85891d82cc"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.175058 28786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:08:05.179075 28801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.180754 28786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:05.180938 28786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "c1f315ac18ed4eebb3095e10b1ce135b"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "36038111543a24195c27cf593c01fa2f"
server_key_iv: "00253e4cab14157a780bde85891d82cc"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.181142 28786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:05.200839 28786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:05.205262 28786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:05.205561 28786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:05.206550 28786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:05.208069 28786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:05.208151 28786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.208271 28786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:05.208328 28786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.240476 28786 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:42021
I20260620 07:08:05.240511 28913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:42021 every 8 connection(s)
I20260620 07:08:05.241786 28786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:05.246598 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28786
I20260620 07:08:05.246829 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:05.247589 11591 external_mini_cluster.cc:1428] Setting key 1c29ab3b7e100e33760de573162bd005
Jun 20 07:08:05 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:05.267354 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:05.244430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:34791 (local address 127.11.81.254:44883)
0620 07:08:05.245007 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:05.245013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:05.246002 (+ 989us) server_negotiation.cc:407] Connection header received
0620 07:08:05.249002 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:05.249007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:05.249166 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:05.249287 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:05.251672 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.252777 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.255188 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.255782 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.255919 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:05.260837 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:05.260843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:05.260847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:05.260888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:05.263779 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:05.264788 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:05.264792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:05.264795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:05.264944 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:05.265424 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:05.265427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:05.265429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:05.265715 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:05.265860 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:05.266674 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 07:08:05.266937 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260620 07:08:05.268389 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:05.245020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44883 (local address 127.11.81.193:34791)
0620 07:08:05.245748 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:05.245795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:05.248348 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:05.249649 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:05.249662 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:05.250329 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:05.251270 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:05.251340 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.252977 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.252983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:05.254918 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:05.254940 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.256204 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.256215 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:05.256555 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:05.257417 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:05.257455 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:05.260546 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:05.263974 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:05.263990 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:05.264006 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:05.264632 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:05.265068 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:05.265073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:05.265077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:05.265295 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:05.265860 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:05.265869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:05.266262 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 07:08:05.266534 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 07:08:05.267145 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":269,"threads_started":1}
I20260620 07:08:05.270107 28914 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44883
I20260620 07:08:05.270438 28914 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:05.271193 28914 heartbeater.cc:511] Master 127.11.81.254:44883 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:08:05.273547 28725 ts_manager.cc:194] Registered new tserver with Master: c1f315ac18ed4eebb3095e10b1ce135b (127.11.81.193:42021)
I20260620 07:08:05.275128 28725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:34791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:08:05.302567 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:05.642581 28920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:05.642882 28920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:05.642975 28920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:05.650357 28920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:05.650486 28920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:05.650602 28920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:05.660820 28920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:05.662261 28920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:05.663659 28920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:05.676909 28925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:05.676988 28926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:05.676880 28928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:05.677608 28920 server_base.cc:1034] running on GCE node
I20260620 07:08:05.678278 28920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:05.679436 28920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:05.680660 28920 hybrid_clock.cc:648] HybridClock initialized: now 1781939285680602 us; error 50 us; skew 500 ppm
I20260620 07:08:05.681066 28920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:05 dist-test-slave-wvz3 krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:05.684404 28920 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:08:05.686153 28920 webserver.cc:460] Webserver started at http://127.11.81.194:45203/ using document root <none> and password file <none>
I20260620 07:08:05.686857 28920 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:05.686993 28920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:05.687325 28920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:05.690380 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "ade9bd9efccd43d4887b6d863d397ec8"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "4a51007a3d1f0e0532432e9a63351ee5"
server_key_iv: "6ba14ef19fd5f8cac7b588a7b2bb2090"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.691144 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "ade9bd9efccd43d4887b6d863d397ec8"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "4a51007a3d1f0e0532432e9a63351ee5"
server_key_iv: "6ba14ef19fd5f8cac7b588a7b2bb2090"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.696928 28920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:08:05.700753 28935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.702395 28920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:05.702565 28920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "ade9bd9efccd43d4887b6d863d397ec8"
format_stamp: "Formatted at 2026-06-20 07:08:05 on dist-test-slave-wvz3"
server_key: "4a51007a3d1f0e0532432e9a63351ee5"
server_key_iv: "6ba14ef19fd5f8cac7b588a7b2bb2090"
server_key_version: "encryptionkey@0"
I20260620 07:08:05.702754 28920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:05.728096 28920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:05.732844 28920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:05.733129 28920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:05.734136 28920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:05.735570 28920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:05.735649 28920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.735774 28920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:05.735824 28920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:05.767256 28920 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:41609
I20260620 07:08:05.767357 29047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:41609 every 8 connection(s)
I20260620 07:08:05.768623 28920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:05.776662 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 28920
I20260620 07:08:05.776829 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:05.777297 11591 external_mini_cluster.cc:1428] Setting key 607b2a501735242f186904b0491f34cf
Jun 20 07:08:05 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939285, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:05.790648 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:05.771214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:41473 (local address 127.11.81.254:44883)
0620 07:08:05.771660 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:05.771664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:05.772640 (+ 976us) server_negotiation.cc:407] Connection header received
0620 07:08:05.774745 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:05.774752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:05.774871 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:05.774971 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:05.776653 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.777475 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.779228 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.779935 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.780078 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:05.783968 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:05.783976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:05.783981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:05.784026 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:05.787198 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:05.788115 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:05.788122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:05.788126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:05.788306 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:05.788780 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:05.788785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:05.788788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:05.789133 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:05.789253 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:05.790169 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 07:08:05.790376 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
I20260620 07:08:05.792085 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:05.771767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44883 (local address 127.11.81.194:41473)
0620 07:08:05.772471 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:05.772519 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:05.774424 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:05.775233 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:05.775241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:05.775838 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:05.776469 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:05.776489 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.777624 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.777628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:05.779047 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:05.779064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:05.780177 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:05.780182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:05.780428 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:05.781178 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:05.781203 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:05.783692 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:05.787429 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:05.787438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:05.787454 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:05.787945 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:05.788440 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:05.788444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:05.788446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:05.788658 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:05.789269 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:05.789278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:05.789763 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 07:08:05.790062 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 07:08:05.790631 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":266,"threads_started":1}
I20260620 07:08:05.793536 29048 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44883
I20260620 07:08:05.793865 29048 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:05.794727 29048 heartbeater.cc:511] Master 127.11.81.254:44883 requested a full tablet report, sending...
I20260620 07:08:05.796402 28725 ts_manager.cc:194] Registered new tserver with Master: ade9bd9efccd43d4887b6d863d397ec8 (127.11.81.194:41609)
I20260620 07:08:05.797253 28725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:41473
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:08:05.834717 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:06.162997 29054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:06.163367 29054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:06.163511 29054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:06.171001 29054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:06.171154 29054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:06.171357 29054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:06.181607 29054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:06.183027 29054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:06.184548 29054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:06.198556 29059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:06.198529 29060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:06.198580 29062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:06.199249 29054 server_base.cc:1034] running on GCE node
I20260620 07:08:06.199939 29054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:06.200999 29054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:06.202201 29054 hybrid_clock.cc:648] HybridClock initialized: now 1781939286202140 us; error 56 us; skew 500 ppm
I20260620 07:08:06.202533 29054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:06 dist-test-slave-wvz3 krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939286, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:06.205751 29054 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:08:06.207532 29054 webserver.cc:460] Webserver started at http://127.11.81.195:45229/ using document root <none> and password file <none>
I20260620 07:08:06.208247 29054 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:06.208365 29054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:06.208693 29054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:06.211941 29054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "cfe2c42a0b8447f2847a11050e88ad49"
format_stamp: "Formatted at 2026-06-20 07:08:06 on dist-test-slave-wvz3"
server_key: "d766c363d3be668732dfc0476dedd5ab"
server_key_iv: "9a402abe582c7123086f72f511d7a1e8"
server_key_version: "encryptionkey@0"
I20260620 07:08:06.212770 29054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "cfe2c42a0b8447f2847a11050e88ad49"
format_stamp: "Formatted at 2026-06-20 07:08:06 on dist-test-slave-wvz3"
server_key: "d766c363d3be668732dfc0476dedd5ab"
server_key_iv: "9a402abe582c7123086f72f511d7a1e8"
server_key_version: "encryptionkey@0"
I20260620 07:08:06.218540 29054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:08:06.222499 29069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:06.224110 29054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:06.224291 29054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "cfe2c42a0b8447f2847a11050e88ad49"
format_stamp: "Formatted at 2026-06-20 07:08:06 on dist-test-slave-wvz3"
server_key: "d766c363d3be668732dfc0476dedd5ab"
server_key_iv: "9a402abe582c7123086f72f511d7a1e8"
server_key_version: "encryptionkey@0"
I20260620 07:08:06.224499 29054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:06.248334 29054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:06.253005 29054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:06.253304 29054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:06.254320 29054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:06.255775 29054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:06.255859 29054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:06.255985 29054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:06.256047 29054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:06.278582 28914 heartbeater.cc:503] Master 127.11.81.254:44883 was elected leader, sending a full tablet report...
I20260620 07:08:06.287345 29054 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:41043
I20260620 07:08:06.287441 29181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:41043 every 8 connection(s)
I20260620 07:08:06.288657 29054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:06.299043 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29054
I20260620 07:08:06.299216 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:06.299664 11591 external_mini_cluster.cc:1428] Setting key fd4ce949f9944cad18f5ea6d47c7ff81
Jun 20 07:08:06 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939286, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:06.308993 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.291335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:40867 (local address 127.11.81.254:44883)
0620 07:08:06.291791 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.291796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.292899 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 07:08:06.294840 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.294846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.294946 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.295045 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:06.296656 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.297428 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.299013 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.299504 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.299621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.303361 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:06.303366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:06.303369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:06.303406 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:06.306050 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.306833 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.306837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.306839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.306964 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.307389 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.307392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.307394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.307690 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:06.307827 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.308539 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.308716 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":219,"threads_started":1}
I20260620 07:08:06.309957 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.291921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44883 (local address 127.11.81.195:40867)
0620 07:08:06.292700 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.292746 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.294539 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.295338 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.295351 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.295897 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:06.296476 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.296495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.297596 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.297600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.298866 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.298879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.299734 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.299741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.299982 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.300667 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:06.300699 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:06.303131 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:06.306189 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:06.306196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:06.306212 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:06.306742 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:06.307055 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:06.307058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:06.307060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:06.307251 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:06.307815 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:06.307821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:06.308177 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.308413 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.308822 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":310,"threads_started":1}
I20260620 07:08:06.311493 29182 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44883
I20260620 07:08:06.311877 29182 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:06.312608 29182 heartbeater.cc:511] Master 127.11.81.254:44883 requested a full tablet report, sending...
I20260620 07:08:06.314097 28725 ts_manager.cc:194] Registered new tserver with Master: cfe2c42a0b8447f2847a11050e88ad49 (127.11.81.195:41043)
I20260620 07:08:06.315038 28725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:40867
I20260620 07:08:06.315156 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:06.332654 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.319700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.11.81.254:44883)
0620 07:08:06.319933 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.319938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.320245 (+ 307us) server_negotiation.cc:407] Connection header received
0620 07:08:06.320392 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.320396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.320471 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.320558 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:06.321952 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.322680 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.324038 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.324435 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.324533 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.326931 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:06.326938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:06.326942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:06.326981 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:06.329826 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.330608 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.330613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.330615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.330763 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.331165 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.331169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.331171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.331586 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:06.331778 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.332203 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.332418 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:08:06.336972 28725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:06.343886 28725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:06.378172 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.356936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:41609 (local address 127.0.0.1:47700)
0620 07:08:06.367584 (+ 10648us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.367650 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.367904 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.369181 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.369190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.369239 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:06.369656 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.369669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.372508 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.372513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.374832 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.374846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.377131 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.377139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.377397 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.377451 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.377591 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.377784 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10273,"thread_start_us":14544,"threads_started":1}
I20260620 07:08:06.378295 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.357183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.11.81.194:41609)
0620 07:08:06.367969 (+ 10786us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.367979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.368012 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:08:06.368127 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.368137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.368448 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.368670 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:06.369911 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.372315 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.375025 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.376536 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.376671 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.376890 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.377660 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.377942 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10615,"thread_start_us":11688,"threads_started":1}
I20260620 07:08:06.382820 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.356984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:41043 (local address 127.0.0.1:56150)
0620 07:08:06.371749 (+ 14765us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.371769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.371889 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.374226 (+ 2337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.374232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.374271 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:06.374708 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.376074 (+ 1366us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.378001 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.378006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.380196 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.380207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.382004 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.382010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.382185 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.382218 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.382320 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.382463 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14636,"thread_start_us":10026,"threads_started":1}
I20260620 07:08:06.382478 28983 tablet_service.cc:1467] Processing CreateTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 (DEFAULT_TABLE table=test-table [id=1fc3f2598a18401a9c907422df83cf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:06.382891 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.357040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.11.81.195:41043)
0620 07:08:06.373301 (+ 16261us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.373310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.373347 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:08:06.373447 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.373457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.373746 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.373961 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:06.376282 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.377800 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.380359 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.381762 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.381895 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.382130 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.382400 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.382566 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16079,"thread_start_us":15956,"threads_started":1}
I20260620 07:08:06.384191 28983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30e7fb3781f45dd8a6e7605de01bdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:06.384996 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.373469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:42021 (local address 127.0.0.1:42540)
0620 07:08:06.377197 (+ 3728us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.377213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.377345 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.378263 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.378268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.378295 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:06.378637 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.378647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.380518 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.380523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.382437 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.382451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.384270 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.384276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.384455 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.384487 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.384570 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.384677 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3578,"thread_start_us":3350,"threads_started":1}
I20260620 07:08:06.385564 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.368645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.11.81.193:42021)
0620 07:08:06.376788 (+ 8143us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.376798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.377334 (+ 536us) server_negotiation.cc:407] Connection header received
0620 07:08:06.377493 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.377499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.377761 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.377977 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:06.378840 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.380383 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.382620 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.384072 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.384194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.384411 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.385059 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.385230 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7968,"thread_start_us":7806,"threads_started":1}
I20260620 07:08:06.387862 29117 tablet_service.cc:1467] Processing CreateTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 (DEFAULT_TABLE table=test-table [id=1fc3f2598a18401a9c907422df83cf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:06.388543 28849 tablet_service.cc:1467] Processing CreateTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 (DEFAULT_TABLE table=test-table [id=1fc3f2598a18401a9c907422df83cf67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:06.389411 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30e7fb3781f45dd8a6e7605de01bdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:06.389844 28849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e30e7fb3781f45dd8a6e7605de01bdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:06.400961 29201 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Bootstrap starting.
I20260620 07:08:06.401897 29202 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Bootstrap starting.
I20260620 07:08:06.405408 29203 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Bootstrap starting.
I20260620 07:08:06.405725 29202 tablet_bootstrap.cc:654] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:06.405797 29201 tablet_bootstrap.cc:654] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:06.407063 29202 log.cc:824] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:06.407115 29201 log.cc:824] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:06.409046 29203 tablet_bootstrap.cc:654] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:06.410146 29201 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: No bootstrap required, opened a new log
I20260620 07:08:06.410352 29202 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: No bootstrap required, opened a new log
I20260620 07:08:06.410305 29203 log.cc:824] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:06.410413 29201 ts_tablet_manager.cc:1397] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 07:08:06.410625 29202 ts_tablet_manager.cc:1397] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:08:06.413586 29203 tablet_bootstrap.cc:492] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: No bootstrap required, opened a new log
I20260620 07:08:06.413893 29203 ts_tablet_manager.cc:1397] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 07:08:06.418807 29202 raft_consensus.cc:348] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.419224 29202 raft_consensus.cc:374] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:06.419358 29202 raft_consensus.cc:729] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f315ac18ed4eebb3095e10b1ce135b, State: Initialized, Role: FOLLOWER
I20260620 07:08:06.420007 29202 consensus_queue.cc:260] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.420951 29201 raft_consensus.cc:348] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.421463 29201 raft_consensus.cc:374] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:06.421587 29201 raft_consensus.cc:729] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade9bd9efccd43d4887b6d863d397ec8, State: Initialized, Role: FOLLOWER
I20260620 07:08:06.421921 29202 ts_tablet_manager.cc:1428] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 07:08:06.422379 29201 consensus_queue.cc:260] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.424695 29203 raft_consensus.cc:348] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.425225 29203 raft_consensus.cc:374] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:06.425366 29203 raft_consensus.cc:729] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe2c42a0b8447f2847a11050e88ad49, State: Initialized, Role: FOLLOWER
I20260620 07:08:06.425598 29048 heartbeater.cc:503] Master 127.11.81.254:44883 was elected leader, sending a full tablet report...
I20260620 07:08:06.425988 29203 consensus_queue.cc:260] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.426349 29201 ts_tablet_manager.cc:1428] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 07:08:06.428059 29182 heartbeater.cc:503] Master 127.11.81.254:44883 was elected leader, sending a full tablet report...
I20260620 07:08:06.428408 29203 ts_tablet_manager.cc:1428] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
W20260620 07:08:06.494230 28915 tablet.cc:2369] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:06.520619 29049 tablet.cc:2369] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:06.540330 29183 tablet.cc:2369] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:06.741233 29208 raft_consensus.cc:482] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:06.741545 29208 raft_consensus.cc:504] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.743196 29208 leader_election.cc:288] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe2c42a0b8447f2847a11050e88ad49 (127.11.81.195:41043), c1f315ac18ed4eebb3095e10b1ce135b (127.11.81.193:42021)
I20260620 07:08:06.750051 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.743600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:46847 (local address 127.11.81.195:41043)
0620 07:08:06.743760 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.743765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.744354 (+ 589us) server_negotiation.cc:407] Connection header received
0620 07:08:06.744736 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.744741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.744828 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.744986 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:06.745564 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.746296 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.748254 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.749041 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.749190 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.749272 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.749619 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.749787 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:08:06.750120 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.744561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:44107 (local address 127.11.81.193:42021)
0620 07:08:06.744851 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.744857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.745094 (+ 237us) server_negotiation.cc:407] Connection header received
0620 07:08:06.745346 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.745349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.745406 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.745477 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:06.745980 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.746667 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.748409 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.749190 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.749338 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.749419 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.749743 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.749886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:08:06.750202 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.743569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:41043 (local address 127.11.81.194:46847)
0620 07:08:06.744213 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.744232 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.744378 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.745027 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.745030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.745059 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:06.745410 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.745421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.746431 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.746436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.748091 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.748104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.749245 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.749251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.749458 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.749485 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.749575 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.749715 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":59,"thread_start_us":308,"threads_started":1}
I20260620 07:08:06.750270 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.744386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:42021 (local address 127.11.81.194:44107)
0620 07:08:06.744988 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:06.745005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:06.745205 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:06.745509 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:06.745514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:06.745537 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:06.745841 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.745850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.746801 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.746809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.748250 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:06.748262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.749407 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.749413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:06.749592 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.749617 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:08:06.749708 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:08:06.749851 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":21,"thread_start_us":274,"threads_started":1}
I20260620 07:08:06.751245 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30e7fb3781f45dd8a6e7605de01bdc9" candidate_uuid: "ade9bd9efccd43d4887b6d863d397ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe2c42a0b8447f2847a11050e88ad49" is_pre_election: true
I20260620 07:08:06.751243 28869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30e7fb3781f45dd8a6e7605de01bdc9" candidate_uuid: "ade9bd9efccd43d4887b6d863d397ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" is_pre_election: true
I20260620 07:08:06.751748 29137 raft_consensus.cc:2454] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ade9bd9efccd43d4887b6d863d397ec8 in term 0.
I20260620 07:08:06.751792 28869 raft_consensus.cc:2454] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ade9bd9efccd43d4887b6d863d397ec8 in term 0.
I20260620 07:08:06.752360 28938 leader_election.cc:302] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ade9bd9efccd43d4887b6d863d397ec8, cfe2c42a0b8447f2847a11050e88ad49; no voters:
I20260620 07:08:06.752738 29208 raft_consensus.cc:2781] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:06.752905 29208 raft_consensus.cc:482] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:06.753021 29208 raft_consensus.cc:3037] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:06.755203 29208 raft_consensus.cc:504] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.755856 29208 leader_election.cc:288] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [CANDIDATE]: Term 1 election: Requested vote from peers cfe2c42a0b8447f2847a11050e88ad49 (127.11.81.195:41043), c1f315ac18ed4eebb3095e10b1ce135b (127.11.81.193:42021)
I20260620 07:08:06.756446 29137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30e7fb3781f45dd8a6e7605de01bdc9" candidate_uuid: "ade9bd9efccd43d4887b6d863d397ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe2c42a0b8447f2847a11050e88ad49"
I20260620 07:08:06.756459 28869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e30e7fb3781f45dd8a6e7605de01bdc9" candidate_uuid: "ade9bd9efccd43d4887b6d863d397ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f315ac18ed4eebb3095e10b1ce135b"
I20260620 07:08:06.756651 29137 raft_consensus.cc:3037] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:06.756639 28869 raft_consensus.cc:3037] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:06.758888 28869 raft_consensus.cc:2454] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ade9bd9efccd43d4887b6d863d397ec8 in term 1.
I20260620 07:08:06.758906 29137 raft_consensus.cc:2454] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ade9bd9efccd43d4887b6d863d397ec8 in term 1.
I20260620 07:08:06.759335 28938 leader_election.cc:302] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ade9bd9efccd43d4887b6d863d397ec8, c1f315ac18ed4eebb3095e10b1ce135b; no voters:
I20260620 07:08:06.759662 29208 raft_consensus.cc:2781] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:06.760116 29208 raft_consensus.cc:686] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 1 LEADER]: Becoming Leader. State: Replica: ade9bd9efccd43d4887b6d863d397ec8, State: Running, Role: LEADER
I20260620 07:08:06.760679 29208 consensus_queue.cc:237] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } }
I20260620 07:08:06.765403 28724 catalog_manager.cc:5495] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ade9bd9efccd43d4887b6d863d397ec8 (127.11.81.194). New cstate: current_term: 1 leader_uuid: "ade9bd9efccd43d4887b6d863d397ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ade9bd9efccd43d4887b6d863d397ec8" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 41609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:06.842038 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.834822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.11.81.194:41609)
0620 07:08:06.835064 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.835070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.835337 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:08:06.835535 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.835539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.835621 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.835759 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:06.836181 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.837215 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.839261 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.839924 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.840061 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.840666 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:06.841417 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:06.841570 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.841646 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.841742 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:08:06.851127 28869 raft_consensus.cc:1261] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 1 FOLLOWER]: Refusing update from remote peer ade9bd9efccd43d4887b6d863d397ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:06.851127 29137 raft_consensus.cc:1261] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 1 FOLLOWER]: Refusing update from remote peer ade9bd9efccd43d4887b6d863d397ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:06.852136 29212 consensus_queue.cc:1035] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe2c42a0b8447f2847a11050e88ad49" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:06.852492 29208 consensus_queue.cc:1035] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f315ac18ed4eebb3095e10b1ce135b" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:06.863592 29216 mvcc.cc:205] Tried to move back new op lower bound from 7298823318931931136 to 7298823318576005120. Current Snapshot: MvccSnapshot[applied={T|T < 7298823318931931136}]
I20260620 07:08:06.872694 29218 mvcc.cc:205] Tried to move back new op lower bound from 7298823318931931136 to 7298823318576005120. Current Snapshot: MvccSnapshot[applied={T|T < 7298823318931931136}]
I20260620 07:08:06.886363 28725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:06.886615 28725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:06.890616 28725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9514ece5b8e4aa8bc153355a5cea451: Sending DeleteTablet for 3 replicas of tablet e30e7fb3781f45dd8a6e7605de01bdc9
I20260620 07:08:06.892004 28983 tablet_service.cc:1514] Processing DeleteTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:06 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:47700
I20260620 07:08:06.892055 28849 tablet_service.cc:1514] Processing DeleteTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:06 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:42540
I20260620 07:08:06.893098 29117 tablet_service.cc:1514] Processing DeleteTablet for tablet e30e7fb3781f45dd8a6e7605de01bdc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:06 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:56150
I20260620 07:08:06.893898 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28786
I20260620 07:08:06.899822 29226 tablet_replica.cc:330] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: stopping tablet replica
I20260620 07:08:06.900349 29226 raft_consensus.cc:2229] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:06.900770 29228 tablet_replica.cc:330] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: stopping tablet replica
I20260620 07:08:06.900954 29226 raft_consensus.cc:2258] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:06.901180 29228 raft_consensus.cc:2229] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:06.901669 29228 raft_consensus.cc:2258] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:06.902514 29227 tablet_replica.cc:330] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: stopping tablet replica
I20260620 07:08:06.902972 29227 raft_consensus.cc:2229] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:06.903738 29227 raft_consensus.cc:2258] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:06.904345 29228 ts_tablet_manager.cc:1905] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:06.905640 29226 ts_tablet_manager.cc:1905] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:06.905790 29227 ts_tablet_manager.cc:1905] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:06.911382 29228 ts_tablet_manager.cc:1918] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:06.911662 29228 log.cc:1196] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/e30e7fb3781f45dd8a6e7605de01bdc9
I20260620 07:08:06.912204 29228 ts_tablet_manager.cc:1939] T e30e7fb3781f45dd8a6e7605de01bdc9 P cfe2c42a0b8447f2847a11050e88ad49: Deleting consensus metadata
I20260620 07:08:06.913419 29227 ts_tablet_manager.cc:1918] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:06.913632 29227 log.cc:1196] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/e30e7fb3781f45dd8a6e7605de01bdc9
I20260620 07:08:06.914139 29227 ts_tablet_manager.cc:1939] T e30e7fb3781f45dd8a6e7605de01bdc9 P ade9bd9efccd43d4887b6d863d397ec8: Deleting consensus metadata
I20260620 07:08:06.914233 28711 catalog_manager.cc:4841] TS cfe2c42a0b8447f2847a11050e88ad49 (127.11.81.195:41043): tablet e30e7fb3781f45dd8a6e7605de01bdc9 (table test-table [id=1fc3f2598a18401a9c907422df83cf67]) successfully deleted
I20260620 07:08:06.916138 28709 catalog_manager.cc:4841] TS ade9bd9efccd43d4887b6d863d397ec8 (127.11.81.194:41609): tablet e30e7fb3781f45dd8a6e7605de01bdc9 (table test-table [id=1fc3f2598a18401a9c907422df83cf67]) successfully deleted
I20260620 07:08:06.917147 29226 ts_tablet_manager.cc:1918] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:06.917338 29226 log.cc:1196] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/e30e7fb3781f45dd8a6e7605de01bdc9
I20260620 07:08:06.917807 29226 ts_tablet_manager.cc:1939] T e30e7fb3781f45dd8a6e7605de01bdc9 P c1f315ac18ed4eebb3095e10b1ce135b: Deleting consensus metadata
Jun 20 07:08:06 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:08:06.919450 28711 catalog_manager.cc:4841] TS c1f315ac18ed4eebb3095e10b1ce135b (127.11.81.193:42021): tablet e30e7fb3781f45dd8a6e7605de01bdc9 (table test-table [id=1fc3f2598a18401a9c907422df83cf67]) successfully deleted
I20260620 07:08:06.925750 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:06.895376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.11.81.193:42021)
0620 07:08:06.895570 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:06.895578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:06.900130 (+ 4552us) server_negotiation.cc:407] Connection header received
0620 07:08:06.900318 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:06.900324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:06.900422 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:06.900532 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:06.902607 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.903726 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.912507 (+ 8781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:06.913132 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:06.913272 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:06.919526 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:06.919542 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:06.919561 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:06.919607 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:06.922541 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.923588 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.923596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.923599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.923764 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:06.924267 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:06.924270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:06.924272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:06.924628 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:06.924904 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:06.925259 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 07:08:06.925466 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:08:06.926620 28909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:07.019691 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28786
I20260620 07:08:07.035984 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28920
Jun 20 07:08:07 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:08:07.054831 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:07.036786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.11.81.194:41609)
0620 07:08:07.036955 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:07.036959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:07.036978 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:08:07.037175 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:07.037180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:07.037244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:07.037328 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:07.039015 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.040047 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.041889 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.042378 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.042499 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:07.048137 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:07.048157 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:07.048174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:07.048222 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:07.051491 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:07.052470 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:07.052478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:07.052480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:07.052643 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:07.053083 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:07.053087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:07.053089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:07.053510 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:07.053779 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:07.054224 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 07:08:07.054424 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:08:07.055584 29043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:07.169844 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28920
I20260620 07:08:07.185139 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29054
Jun 20 07:08:07 dist-test-slave-wvz3 krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:08:07.202250 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:07.185970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.11.81.195:41043)
0620 07:08:07.186183 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:07.186188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:07.186210 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:08:07.186367 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:07.186371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:07.186437 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:07.186520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:07.188052 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.188784 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.190226 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.190633 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.190749 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:07.195553 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:07.195574 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:07.195590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:07.195641 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:07.198643 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:07.199743 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:07.199752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:07.199754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:07.200009 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:07.200481 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:07.200485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:07.200488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:07.200974 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:07.201213 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:07.201629 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 07:08:07.201896 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 07:08:07.203011 29177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:07.292030 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29054
I20260620 07:08:07.305332 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 28693
I20260620 07:08:07.306423 28755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:07.416314 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 28693
2026-06-20T07:08:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4545 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T07:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:07Z Disabled control of system clock
I20260620 07:08:07.494884 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:33153
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:33153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:07.833209 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:07.833560 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:07.833681 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:07.841423 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:07.841567 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:07.841619 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:07.841720 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:07.841792 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:07.851619 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:33153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:33153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:07.853014 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:07.854910 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:07.866825 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:07.866866 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:07.867466 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:07.868161 29237 server_base.cc:1034] running on GCE node
I20260620 07:08:07.868957 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:07.870474 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:07.871688 29237 hybrid_clock.cc:648] HybridClock initialized: now 1781939287871644 us; error 29 us; skew 500 ppm
I20260620 07:08:07.872073 29237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:07.874423 29237 webserver.cc:460] Webserver started at http://127.11.81.254:44187/ using document root <none> and password file <none>
I20260620 07:08:07.875109 29237 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:07.875196 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:07.875526 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:07.878384 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "e87b0483a25e4462a2349af55f763fdd"
format_stamp: "Formatted at 2026-06-20 07:08:07 on dist-test-slave-wvz3"
server_key: "271082f9f4f37c6830720277ee06fd8f"
server_key_iv: "39110ff73d1551058bdb46586730678f"
server_key_version: "encryptionkey@0"
I20260620 07:08:07.879211 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "e87b0483a25e4462a2349af55f763fdd"
format_stamp: "Formatted at 2026-06-20 07:08:07 on dist-test-slave-wvz3"
server_key: "271082f9f4f37c6830720277ee06fd8f"
server_key_iv: "39110ff73d1551058bdb46586730678f"
server_key_version: "encryptionkey@0"
I20260620 07:08:07.884552 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:08:07.888464 29251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:07.890061 29237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:08:07.890240 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "e87b0483a25e4462a2349af55f763fdd"
format_stamp: "Formatted at 2026-06-20 07:08:07 on dist-test-slave-wvz3"
server_key: "271082f9f4f37c6830720277ee06fd8f"
server_key_iv: "39110ff73d1551058bdb46586730678f"
server_key_version: "encryptionkey@0"
I20260620 07:08:07.890460 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:07.911161 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:07.912060 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:07.912334 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:07.929996 29237 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:33153
I20260620 07:08:07.930002 29302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:33153 every 8 connection(s)
I20260620 07:08:07.931382 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:07.935218 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:07.939273 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29237
I20260620 07:08:07.939483 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:07.940028 11591 external_mini_cluster.cc:1428] Setting key 0d3aa8d3ded956421a58285dc42cd7a5
I20260620 07:08:07.946862 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: Bootstrap starting.
I20260620 07:08:07.950186 29303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:07.951229 29303 log.cc:824] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:07.954707 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: No bootstrap required, opened a new log
I20260620 07:08:07.954790 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:07.941982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.11.81.254:33153)
0620 07:08:07.942980 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:07.942993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:07.943035 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:08:07.945461 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:07.945488 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:07.946007 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:07.946390 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:07.946955 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.948310 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.949837 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:07.950601 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:07.950777 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:07.952098 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:07.952105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:07.952120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:07.952722 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:07.952823 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:07.952852 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:07.952966 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:07.953189 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:08:07.953527 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":349,"threads_started":1}
I20260620 07:08:07.962915 29303 raft_consensus.cc:348] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } }
I20260620 07:08:07.963335 29303 raft_consensus.cc:374] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:07.963440 29303 raft_consensus.cc:729] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87b0483a25e4462a2349af55f763fdd, State: Initialized, Role: FOLLOWER
I20260620 07:08:07.964025 29303 consensus_queue.cc:260] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } }
I20260620 07:08:07.964236 29303 raft_consensus.cc:388] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:07.964365 29303 raft_consensus.cc:482] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:07.964514 29303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:07.966887 29303 raft_consensus.cc:504] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } }
I20260620 07:08:07.967453 29303 leader_election.cc:302] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e87b0483a25e4462a2349af55f763fdd; no voters:
I20260620 07:08:07.968026 29303 leader_election.cc:288] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:07.968174 29308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:07.968808 29308 raft_consensus.cc:686] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [term 1 LEADER]: Becoming Leader. State: Replica: e87b0483a25e4462a2349af55f763fdd, State: Running, Role: LEADER
I20260620 07:08:07.969333 29308 consensus_queue.cc:237] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } }
I20260620 07:08:07.970002 29303 sys_catalog.cc:564] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:07.972164 29310 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e87b0483a25e4462a2349af55f763fdd. Latest consensus state: current_term: 1 leader_uuid: "e87b0483a25e4462a2349af55f763fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } } }
I20260620 07:08:07.972272 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e87b0483a25e4462a2349af55f763fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87b0483a25e4462a2349af55f763fdd" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 33153 } } }
I20260620 07:08:07.972579 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:07.972584 29310 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:07.975962 29316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:07.981279 29316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:07.990901 29316 catalog_manager.cc:1347] Generated new cluster ID: 944bb3881f4f4ee981bc0bd770e3ed9b
I20260620 07:08:07.991055 29316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:08.004968 29316 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:08.007138 29316 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:08.022594 29316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: Generated new TSK 0
I20260620 07:08:08.023639 29316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:08.037151 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--builtin_ntp_servers=127.11.81.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:08.370687 29327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:08.371014 29327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:08.371143 29327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:08.378851 29327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:08.378988 29327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:08.379124 29327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:08.388624 29327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:08.389966 29327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:08.391824 29327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:08.405269 29332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:08.405393 29333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:08.406045 29335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:08.406544 29327 server_base.cc:1034] running on GCE node
I20260620 07:08:08.407122 29327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:08.408130 29327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:08.409348 29327 hybrid_clock.cc:648] HybridClock initialized: now 1781939288409282 us; error 50 us; skew 500 ppm
I20260620 07:08:08.409675 29327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:08.412066 29327 webserver.cc:460] Webserver started at http://127.11.81.193:33677/ using document root <none> and password file <none>
I20260620 07:08:08.412801 29327 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:08.412895 29327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:08.413205 29327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:08.416442 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "516150e433cf495c9d88d599e5d6efd8"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "4b24375cb73d506599e4da57d99c70c7"
server_key_iv: "9584db650a5bf8cc1de30360f4d11254"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.417253 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "516150e433cf495c9d88d599e5d6efd8"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "4b24375cb73d506599e4da57d99c70c7"
server_key_iv: "9584db650a5bf8cc1de30360f4d11254"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.422703 29327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:08:08.426411 29341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.427891 29327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:08.428049 29327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "516150e433cf495c9d88d599e5d6efd8"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "4b24375cb73d506599e4da57d99c70c7"
server_key_iv: "9584db650a5bf8cc1de30360f4d11254"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.428259 29327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:08.451363 29327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:08.452301 29327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:08.452615 29327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:08.453768 29327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:08.455505 29327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:08.455596 29327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.455715 29327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:08.455787 29327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.487757 29327 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:36951
I20260620 07:08:08.487862 29453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:36951 every 8 connection(s)
I20260620 07:08:08.489100 29327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:08.491387 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29327
I20260620 07:08:08.491600 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:08.492084 11591 external_mini_cluster.cc:1428] Setting key 610e1d769d177a4fb3cef07df3b65aed
I20260620 07:08:08.495388 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--builtin_ntp_servers=127.11.81.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:08.504503 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:08.492163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:33383 (local address 127.11.81.254:33153)
0620 07:08:08.492682 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:08.492688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:08.494099 (+ 1411us) server_negotiation.cc:407] Connection header received
0620 07:08:08.496598 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:08.496603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:08.496679 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:08.496778 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:08.498301 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.499121 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.501433 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.501839 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.501970 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:08.503600 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:08.503605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:08.503609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:08.503630 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:08.503683 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:08.503696 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:08.503768 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:08.504027 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:08:08.504143 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260620 07:08:08.505460 29457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:08.492871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:33153 (local address 127.11.81.193:33383)
0620 07:08:08.493924 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:08.493981 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:08.496242 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:08.497078 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:08.497089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:08.497229 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:08.498061 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:08.498087 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.499341 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.499347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:08.501270 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:08.501288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.502070 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.502077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:08.502378 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:08.503180 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:08.503184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:08.503242 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:08.503403 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:08.503793 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:08.503806 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:08.503815 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:08:08.503973 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 07:08:08.504319 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":315,"threads_started":1}
I20260620 07:08:08.507093 29454 heartbeater.cc:347] Connected to a master server at 127.11.81.254:33153
I20260620 07:08:08.507439 29454 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:08.508230 29454 heartbeater.cc:511] Master 127.11.81.254:33153 requested a full tablet report, sending...
I20260620 07:08:08.510491 29268 ts_manager.cc:194] Registered new tserver with Master: 516150e433cf495c9d88d599e5d6efd8 (127.11.81.193:36951)
I20260620 07:08:08.512136 29268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.193:33383
W20260620 07:08:08.829851 29458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:08.830155 29458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:08.830261 29458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:08.838599 29458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:08.838742 29458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:08.838922 29458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:08.851990 29458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:08.853523 29458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:08.856029 29458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:08.872567 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:08.872504 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:08.872891 29466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:08.874553 29458 server_base.cc:1034] running on GCE node
I20260620 07:08:08.875386 29458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:08.876715 29458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:08.877981 29458 hybrid_clock.cc:648] HybridClock initialized: now 1781939288877913 us; error 48 us; skew 500 ppm
I20260620 07:08:08.878389 29458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:08.881479 29458 webserver.cc:460] Webserver started at http://127.11.81.194:44057/ using document root <none> and password file <none>
I20260620 07:08:08.882498 29458 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:08.882627 29458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:08.883031 29458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:08.886407 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "a775da89ce994eb2a9686719e9f69132"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "7edaf6b4a8a303a90fdffeec68afbf71"
server_key_iv: "18b467369532eecc531f8b19e9c1407f"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.887216 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "a775da89ce994eb2a9686719e9f69132"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "7edaf6b4a8a303a90fdffeec68afbf71"
server_key_iv: "18b467369532eecc531f8b19e9c1407f"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.892860 29458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:08:08.896903 29472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.898331 29458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:08.898476 29458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "a775da89ce994eb2a9686719e9f69132"
format_stamp: "Formatted at 2026-06-20 07:08:08 on dist-test-slave-wvz3"
server_key: "7edaf6b4a8a303a90fdffeec68afbf71"
server_key_iv: "18b467369532eecc531f8b19e9c1407f"
server_key_version: "encryptionkey@0"
I20260620 07:08:08.898669 29458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:08.924767 29458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:08.925596 29458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:08.925922 29458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:08.926898 29458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:08.928369 29458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:08.928469 29458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.928575 29458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:08.928643 29458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:08.962589 29458 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:33077
I20260620 07:08:08.962641 29584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:33077 every 8 connection(s)
I20260620 07:08:08.964040 29458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:08.972867 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29458
I20260620 07:08:08.973102 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:08.973773 11591 external_mini_cluster.cc:1428] Setting key 54f0dc9e8289298325f5d4c64285955b
I20260620 07:08:08.977201 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--builtin_ntp_servers=127.11.81.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:08.978250 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:08.966606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:53823 (local address 127.11.81.254:33153)
0620 07:08:08.966863 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:08.966869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:08.968508 (+ 1639us) server_negotiation.cc:407] Connection header received
0620 07:08:08.970488 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:08.970493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:08.970580 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:08.970671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:08.972046 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.972804 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.975063 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.975610 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.975714 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:08.977346 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:08.977354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:08.977359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:08.977386 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:08.977432 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:08.977450 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:08.977557 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:08.977756 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 07:08:08.977875 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:08:08.979225 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:08.967213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:33153 (local address 127.11.81.194:53823)
0620 07:08:08.968298 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:08.968341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:08.970138 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:08.970941 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:08.970950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:08.971057 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:08.971837 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:08.971861 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.973017 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.973023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:08.974937 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:08.974957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:08.975853 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:08.975861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:08.976190 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:08.976951 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:08.976954 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:08.976999 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:08.977087 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:08.977554 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:08.977562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:08.977569 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:08:08.977720 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 07:08:08.978078 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":296,"threads_started":1}
I20260620 07:08:08.980764 29585 heartbeater.cc:347] Connected to a master server at 127.11.81.254:33153
I20260620 07:08:08.981066 29585 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:08.981858 29585 heartbeater.cc:511] Master 127.11.81.254:33153 requested a full tablet report, sending...
I20260620 07:08:08.983258 29268 ts_manager.cc:194] Registered new tserver with Master: a775da89ce994eb2a9686719e9f69132 (127.11.81.194:33077)
I20260620 07:08:08.984145 29268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.194:53823
W20260620 07:08:09.311118 29588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:09.311488 29588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:09.311589 29588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:09.318900 29588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:09.319018 29588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:09.319128 29588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:09.328649 29588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:33153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:09.329908 29588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:09.331835 29588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:09.344295 29593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:09.344411 29596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:09.344447 29594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:09.345539 29588 server_base.cc:1034] running on GCE node
I20260620 07:08:09.346158 29588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:09.347100 29588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:09.348306 29588 hybrid_clock.cc:648] HybridClock initialized: now 1781939289348235 us; error 62 us; skew 500 ppm
I20260620 07:08:09.348625 29588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:09.350965 29588 webserver.cc:460] Webserver started at http://127.11.81.195:40271/ using document root <none> and password file <none>
I20260620 07:08:09.351713 29588 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:09.351801 29588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:09.352084 29588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:09.354948 29588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "20a1421bde6c492085c1834c2f20fb53"
format_stamp: "Formatted at 2026-06-20 07:08:09 on dist-test-slave-wvz3"
server_key: "cfd204c3bc9b19ac88deda45e1de97a4"
server_key_iv: "ffa75d5a2fd1eed345c96dc58a334cef"
server_key_version: "encryptionkey@0"
I20260620 07:08:09.355762 29588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "20a1421bde6c492085c1834c2f20fb53"
format_stamp: "Formatted at 2026-06-20 07:08:09 on dist-test-slave-wvz3"
server_key: "cfd204c3bc9b19ac88deda45e1de97a4"
server_key_iv: "ffa75d5a2fd1eed345c96dc58a334cef"
server_key_version: "encryptionkey@0"
I20260620 07:08:09.360833 29588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:08:09.364466 29602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:09.365831 29588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:08:09.366029 29588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "20a1421bde6c492085c1834c2f20fb53"
format_stamp: "Formatted at 2026-06-20 07:08:09 on dist-test-slave-wvz3"
server_key: "cfd204c3bc9b19ac88deda45e1de97a4"
server_key_iv: "ffa75d5a2fd1eed345c96dc58a334cef"
server_key_version: "encryptionkey@0"
I20260620 07:08:09.366221 29588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:09.386272 29588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:09.387159 29588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:09.387475 29588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:09.388475 29588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:09.390080 29588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:09.390172 29588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:09.390267 29588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:09.390339 29588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:09.420974 29588 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:43969
I20260620 07:08:09.421047 29714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:43969 every 8 connection(s)
I20260620 07:08:09.422304 29588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:09.432494 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29588
I20260620 07:08:09.432672 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:09.433126 11591 external_mini_cluster.cc:1428] Setting key e5f82ee996b13386a2f4f06fcbf4bd8e
I20260620 07:08:09.435139 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.424763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:33523 (local address 127.11.81.254:33153)
0620 07:08:09.424947 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.424952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.426376 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 07:08:09.428385 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.428390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.428479 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.428563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:09.429722 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.430463 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.432242 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.432701 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.432828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.434279 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:09.434287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:09.434292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:09.434317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:09.434358 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:09.434375 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:09.434461 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.434689 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.434831 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:08:09.436439 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.425286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:33153 (local address 127.11.81.195:33523)
0620 07:08:09.426210 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.426253 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.428099 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.428823 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.428832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.428932 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:09.429542 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.429561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.430615 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.430618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.432102 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.432119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.432928 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.432935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.433200 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.433887 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:09.433890 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:09.433935 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:09.434039 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:09.434476 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:09.434486 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:09.434493 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.434645 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.435043 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":256,"threads_started":1}
I20260620 07:08:09.437855 29715 heartbeater.cc:347] Connected to a master server at 127.11.81.254:33153
I20260620 07:08:09.438160 29715 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:09.438932 29715 heartbeater.cc:511] Master 127.11.81.254:33153 requested a full tablet report, sending...
I20260620 07:08:09.440487 29268 ts_manager.cc:194] Registered new tserver with Master: 20a1421bde6c492085c1834c2f20fb53 (127.11.81.195:43969)
I20260620 07:08:09.441335 29268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.195:33523
I20260620 07:08:09.448573 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:09.458515 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.452657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.11.81.254:33153)
0620 07:08:09.452902 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.452909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.453192 (+ 283us) server_negotiation.cc:407] Connection header received
0620 07:08:09.453394 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.453398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.453478 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.453558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:09.454116 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.454898 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.456350 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.456794 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.456893 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.457918 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:09.457923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:09.457927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:09.457947 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:09.457979 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:09.457991 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:09.458060 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.458143 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.458225 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:08:09.462280 29268 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:09.469409 29268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:09.506459 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.490030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:43969 (local address 127.0.0.1:60188)
0620 07:08:09.493854 (+ 3824us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.493879 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.493994 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.498694 (+ 4700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.498699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.498729 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:09.499186 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.499200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.501576 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.501581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.503939 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.503953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.505451 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.505458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.505627 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.505688 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.505809 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.506027 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3699,"mutex_wait_us":140,"thread_start_us":7965,"threads_started":1}
I20260620 07:08:09.506760 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.490392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.11.81.193:36951)
0620 07:08:09.497387 (+ 6995us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.497396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.497428 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:08:09.497539 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.497546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.497832 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.498702 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:09.498947 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.500331 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.503333 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.504852 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.504996 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.505307 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.506163 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.506362 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6829,"thread_start_us":6715,"threads_started":1}
I20260620 07:08:09.507099 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.489871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:36951 (local address 127.0.0.1:57428)
0620 07:08:09.492147 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.492208 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.492412 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.498140 (+ 5728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.498153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.498197 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:09.498659 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.498672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.500495 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.500500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.503072 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.503089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.505069 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.505076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.505769 (+ 693us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.505805 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.505887 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.506027 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"mutex_wait_us":1,"thread_start_us":1797,"threads_started":1}
I20260620 07:08:09.507546 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.491752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.11.81.195:43969)
0620 07:08:09.498032 (+ 6280us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.498039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.498073 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:08:09.498186 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.498194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.498462 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.499558 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:09.499818 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.501384 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.504161 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.505217 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.505410 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.505678 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.506899 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.507063 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6075,"thread_start_us":3745,"threads_started":1}
I20260620 07:08:09.507941 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.493305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:33077 (local address 127.0.0.1:55638)
0620 07:08:09.498342 (+ 5037us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.498367 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.498497 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.499148 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.499152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.499178 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:09.499636 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.499648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.501550 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.501555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.503859 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.503877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.505996 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.506002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.506169 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.506201 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.506271 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.506396 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4891,"mutex_wait_us":198,"thread_start_us":729,"threads_started":1}
I20260620 07:08:09.508587 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.493851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.11.81.194:33077)
0620 07:08:09.497750 (+ 3899us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.497761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.498492 (+ 731us) server_negotiation.cc:407] Connection header received
0620 07:08:09.498643 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.498651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.498897 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.499108 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:09.499816 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.501386 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.504115 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.505501 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.505625 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.505865 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.507206 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.507423 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3707,"thread_start_us":3796,"threads_started":1}
I20260620 07:08:09.510947 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 47a389a906624791955de6b2fb7a423f (DEFAULT_TABLE table=test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:09.511152 29389 tablet_service.cc:1467] Processing CreateTablet for tablet 47a389a906624791955de6b2fb7a423f (DEFAULT_TABLE table=test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:09.512511 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a389a906624791955de6b2fb7a423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:09.512144 29520 tablet_service.cc:1467] Processing CreateTablet for tablet 47a389a906624791955de6b2fb7a423f (DEFAULT_TABLE table=test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:09.512987 29389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a389a906624791955de6b2fb7a423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:09.513456 29520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a389a906624791955de6b2fb7a423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:09.515849 29454 heartbeater.cc:503] Master 127.11.81.254:33153 was elected leader, sending a full tablet report...
I20260620 07:08:09.524556 29733 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Bootstrap starting.
I20260620 07:08:09.527737 29734 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Bootstrap starting.
I20260620 07:08:09.529515 29733 tablet_bootstrap.cc:654] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:09.530879 29733 log.cc:824] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:09.531490 29734 tablet_bootstrap.cc:654] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:09.532639 29735 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Bootstrap starting.
I20260620 07:08:09.532747 29734 log.cc:824] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:09.534531 29733 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: No bootstrap required, opened a new log
I20260620 07:08:09.534880 29733 ts_tablet_manager.cc:1397] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 07:08:09.537251 29735 tablet_bootstrap.cc:654] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:09.538470 29735 log.cc:824] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:09.540678 29734 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: No bootstrap required, opened a new log
I20260620 07:08:09.540988 29734 ts_tablet_manager.cc:1397] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260620 07:08:09.542779 29735 tablet_bootstrap.cc:492] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: No bootstrap required, opened a new log
I20260620 07:08:09.543063 29735 ts_tablet_manager.cc:1397] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260620 07:08:09.544963 29733 raft_consensus.cc:348] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.545434 29733 raft_consensus.cc:374] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:09.545533 29733 raft_consensus.cc:729] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a1421bde6c492085c1834c2f20fb53, State: Initialized, Role: FOLLOWER
I20260620 07:08:09.546188 29733 consensus_queue.cc:260] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.547472 29715 heartbeater.cc:503] Master 127.11.81.254:33153 was elected leader, sending a full tablet report...
I20260620 07:08:09.548115 29733 ts_tablet_manager.cc:1428] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 07:08:09.548885 29734 raft_consensus.cc:348] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.549317 29734 raft_consensus.cc:374] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:09.549427 29734 raft_consensus.cc:729] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a775da89ce994eb2a9686719e9f69132, State: Initialized, Role: FOLLOWER
I20260620 07:08:09.550132 29734 consensus_queue.cc:260] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.551450 29585 heartbeater.cc:503] Master 127.11.81.254:33153 was elected leader, sending a full tablet report...
I20260620 07:08:09.552054 29734 ts_tablet_manager.cc:1428] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 07:08:09.552950 29735 raft_consensus.cc:348] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.553418 29735 raft_consensus.cc:374] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:09.553514 29735 raft_consensus.cc:729] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516150e433cf495c9d88d599e5d6efd8, State: Initialized, Role: FOLLOWER
I20260620 07:08:09.554072 29735 consensus_queue.cc:260] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.555709 29735 ts_tablet_manager.cc:1428] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260620 07:08:09.597740 29739 raft_consensus.cc:482] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:09.598060 29739 raft_consensus.cc:504] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.599820 29739 leader_election.cc:288] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516150e433cf495c9d88d599e5d6efd8 (127.11.81.193:36951), a775da89ce994eb2a9686719e9f69132 (127.11.81.194:33077)
I20260620 07:08:09.606104 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.600253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:56049 (local address 127.11.81.194:33077)
0620 07:08:09.600451 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.600457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.600477 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:08:09.600731 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.600735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.600808 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.600886 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:09.601375 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.602304 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.604242 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.605314 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.605463 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.605515 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.605791 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.605904 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 07:08:09.606221 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.600117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:33077 (local address 127.11.81.195:56049)
0620 07:08:09.600405 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.600424 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.600550 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.600982 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.600986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.601016 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:09.601198 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.601209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.602433 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.602439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.604033 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.604045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.605465 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.605471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.605637 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.605673 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.605770 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.605908 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 07:08:09.607096 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.600428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:58687 (local address 127.11.81.193:36951)
0620 07:08:09.600650 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.600655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.601040 (+ 385us) server_negotiation.cc:407] Connection header received
0620 07:08:09.601215 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.601220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.601305 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.601389 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:09.602011 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.602913 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.605078 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.606177 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.606327 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.606390 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.606753 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.606878 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:08:09.607128 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.600223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:36951 (local address 127.11.81.195:58687)
0620 07:08:09.600957 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:09.600974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:09.601092 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:09.601426 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:09.601431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:09.601456 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:09.601794 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.601806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.603082 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.603085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.604886 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:09.604902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.606439 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.606446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:09.606598 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.606621 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:08:09.606701 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:08:09.606821 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":407,"threads_started":1}
I20260620 07:08:09.607209 29540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a389a906624791955de6b2fb7a423f" candidate_uuid: "20a1421bde6c492085c1834c2f20fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a775da89ce994eb2a9686719e9f69132" is_pre_election: true
I20260620 07:08:09.607820 29540 raft_consensus.cc:2454] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a1421bde6c492085c1834c2f20fb53 in term 0.
I20260620 07:08:09.608168 29409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a389a906624791955de6b2fb7a423f" candidate_uuid: "20a1421bde6c492085c1834c2f20fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516150e433cf495c9d88d599e5d6efd8" is_pre_election: true
I20260620 07:08:09.608543 29604 leader_election.cc:302] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a1421bde6c492085c1834c2f20fb53, a775da89ce994eb2a9686719e9f69132; no voters:
I20260620 07:08:09.608799 29409 raft_consensus.cc:2454] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a1421bde6c492085c1834c2f20fb53 in term 0.
I20260620 07:08:09.608966 29739 raft_consensus.cc:2781] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:09.609114 29739 raft_consensus.cc:482] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:09.609220 29739 raft_consensus.cc:3037] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:09.611533 29739 raft_consensus.cc:504] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.612092 29739 leader_election.cc:288] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [CANDIDATE]: Term 1 election: Requested vote from peers 516150e433cf495c9d88d599e5d6efd8 (127.11.81.193:36951), a775da89ce994eb2a9686719e9f69132 (127.11.81.194:33077)
I20260620 07:08:09.612528 29409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a389a906624791955de6b2fb7a423f" candidate_uuid: "20a1421bde6c492085c1834c2f20fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516150e433cf495c9d88d599e5d6efd8"
I20260620 07:08:09.612555 29540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a389a906624791955de6b2fb7a423f" candidate_uuid: "20a1421bde6c492085c1834c2f20fb53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a775da89ce994eb2a9686719e9f69132"
I20260620 07:08:09.612699 29409 raft_consensus.cc:3037] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:09.612735 29540 raft_consensus.cc:3037] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:09.615249 29409 raft_consensus.cc:2454] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a1421bde6c492085c1834c2f20fb53 in term 1.
I20260620 07:08:09.615274 29540 raft_consensus.cc:2454] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a1421bde6c492085c1834c2f20fb53 in term 1.
I20260620 07:08:09.615686 29603 leader_election.cc:302] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a1421bde6c492085c1834c2f20fb53, 516150e433cf495c9d88d599e5d6efd8; no voters:
I20260620 07:08:09.616025 29739 raft_consensus.cc:2781] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:09.616482 29739 raft_consensus.cc:686] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 1 LEADER]: Becoming Leader. State: Replica: 20a1421bde6c492085c1834c2f20fb53, State: Running, Role: LEADER
I20260620 07:08:09.616927 29739 consensus_queue.cc:237] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } }
I20260620 07:08:09.621606 29266 catalog_manager.cc:5495] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a1421bde6c492085c1834c2f20fb53 (127.11.81.195). New cstate: current_term: 1 leader_uuid: "20a1421bde6c492085c1834c2f20fb53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a1421bde6c492085c1834c2f20fb53" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:09.657044 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.651462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.11.81.195:43969)
0620 07:08:09.651646 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.651652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.651822 (+ 170us) server_negotiation.cc:407] Connection header received
0620 07:08:09.652037 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.652041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.652119 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.652277 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:09.652637 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.653375 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.654804 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.655229 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.655385 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.655889 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:09.656449 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:09.656593 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.656669 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.656756 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:08:09.665841 29540 raft_consensus.cc:1261] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 1 FOLLOWER]: Refusing update from remote peer 20a1421bde6c492085c1834c2f20fb53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:09.665841 29409 raft_consensus.cc:1261] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 1 FOLLOWER]: Refusing update from remote peer 20a1421bde6c492085c1834c2f20fb53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:09.666934 29743 consensus_queue.cc:1035] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516150e433cf495c9d88d599e5d6efd8" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:09.667330 29739 consensus_queue.cc:1035] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a775da89ce994eb2a9686719e9f69132" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:08:09.673931 29716 tablet.cc:2369] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:09.680734 29748 mvcc.cc:205] Tried to move back new op lower bound from 7298823330459910144 to 7298823330274648064. Current Snapshot: MvccSnapshot[applied={T|T < 7298823330459910144}]
I20260620 07:08:09.694614 29268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:09.694941 29268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:09.699064 29268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e87b0483a25e4462a2349af55f763fdd: Sending DeleteTablet for 3 replicas of tablet 47a389a906624791955de6b2fb7a423f
I20260620 07:08:09.700404 29520 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a389a906624791955de6b2fb7a423f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:09 UTC) from {username='slave'} at 127.0.0.1:55638
I20260620 07:08:09.700413 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a389a906624791955de6b2fb7a423f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:09 UTC) from {username='slave'} at 127.0.0.1:60188
I20260620 07:08:09.700474 29389 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a389a906624791955de6b2fb7a423f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:09 UTC) from {username='slave'} at 127.0.0.1:57428
I20260620 07:08:09.703200 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29327
I20260620 07:08:09.707763 29757 tablet_replica.cc:330] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: stopping tablet replica
I20260620 07:08:09.708174 29757 raft_consensus.cc:2229] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:09.708680 29757 raft_consensus.cc:2258] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:09.713346 29757 ts_tablet_manager.cc:1905] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:09.720917 29757 ts_tablet_manager.cc:1918] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:09.721237 29757 log.cc:1196] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/47a389a906624791955de6b2fb7a423f
I20260620 07:08:09.721903 29757 ts_tablet_manager.cc:1939] T 47a389a906624791955de6b2fb7a423f P 516150e433cf495c9d88d599e5d6efd8: Deleting consensus metadata
I20260620 07:08:09.724079 29252 catalog_manager.cc:4841] TS 516150e433cf495c9d88d599e5d6efd8 (127.11.81.193:36951): tablet 47a389a906624791955de6b2fb7a423f (table test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]) successfully deleted
I20260620 07:08:09.725348 29756 tablet_replica.cc:330] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: stopping tablet replica
I20260620 07:08:09.725908 29756 raft_consensus.cc:2229] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:09.726769 29756 raft_consensus.cc:2258] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:09.727627 29755 tablet_replica.cc:330] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: stopping tablet replica
I20260620 07:08:09.728053 29755 raft_consensus.cc:2229] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:09.728576 29755 raft_consensus.cc:2258] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:09.729426 29756 ts_tablet_manager.cc:1905] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:09.730552 29755 ts_tablet_manager.cc:1905] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:09.736951 29756 ts_tablet_manager.cc:1918] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:09.737155 29756 log.cc:1196] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/47a389a906624791955de6b2fb7a423f
I20260620 07:08:09.737706 29756 ts_tablet_manager.cc:1939] T 47a389a906624791955de6b2fb7a423f P 20a1421bde6c492085c1834c2f20fb53: Deleting consensus metadata
I20260620 07:08:09.738279 29755 ts_tablet_manager.cc:1918] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:09.738453 29755 log.cc:1196] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/47a389a906624791955de6b2fb7a423f
I20260620 07:08:09.738981 29755 ts_tablet_manager.cc:1939] T 47a389a906624791955de6b2fb7a423f P a775da89ce994eb2a9686719e9f69132: Deleting consensus metadata
I20260620 07:08:09.739724 29254 catalog_manager.cc:4841] TS 20a1421bde6c492085c1834c2f20fb53 (127.11.81.195:43969): tablet 47a389a906624791955de6b2fb7a423f (table test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]) successfully deleted
I20260620 07:08:09.740346 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.704694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.11.81.193:36951)
0620 07:08:09.704918 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.704924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.724658 (+ 19734us) server_negotiation.cc:407] Connection header received
0620 07:08:09.724870 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.724876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.724989 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.725117 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:09.725795 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.727059 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.729009 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.729614 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.729726 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.738097 (+ 8371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:09.738113 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:09.738133 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:09.739187 (+ 1054us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:09.739259 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:09.739335 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:09.739455 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.739761 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.739979 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:08:09.741384 29253 catalog_manager.cc:4841] TS a775da89ce994eb2a9686719e9f69132 (127.11.81.194:33077): tablet 47a389a906624791955de6b2fb7a423f (table test-table [id=d2e3cd5eabd94c99b91728cc8a7f5c29]) successfully deleted
I20260620 07:08:09.741814 29449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:09.831321 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29327
I20260620 07:08:09.846269 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29458
I20260620 07:08:09.854063 29732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.847111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.11.81.194:33077)
0620 07:08:09.847374 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.847380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.847403 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:08:09.847475 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.847480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.847601 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.847725 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:09.848057 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.849239 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.850680 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.851369 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.851509 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.852538 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:09.852550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:09.852570 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:09.853347 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:09.853410 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:09.853435 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:09.853556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.853605 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.853718 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:08:09.854895 29580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:09.943698 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29458
I20260620 07:08:09.958835 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29588
I20260620 07:08:09.966063 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:09.959673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.11.81.195:43969)
0620 07:08:09.959909 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:09.959915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:09.959943 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:08:09.960127 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:09.960136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:09.960214 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:09.960308 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:09.960739 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.961533 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.962898 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:09.963407 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:09.963540 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:09.964628 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:09.964637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:09.964656 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:09.965288 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:09.965338 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:09.965361 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:09.965452 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:09.965598 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:08:09.965699 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:08:09.966876 29710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:10.089789 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29588
I20260620 07:08:10.106032 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29237
I20260620 07:08:10.107097 29298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:10.216490 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29237
2026-06-20T07:08:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:10 dist-test-slave-wvz3 krb5kdc[29765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:10 dist-test-slave-wvz3 krb5kdc[29765](info): set up 2 sockets
Jun 20 07:08:10 dist-test-slave-wvz3 krb5kdc[29765](info): commencing operation
krb5kdc: starting...
W20260620 07:08:11.231238 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:11 dist-test-slave-wvz3 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:11.481312 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44971
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:11.821760 29778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:11.822078 29778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:11.822196 29778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:11.830072 29778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:11.830202 29778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:11.830266 29778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:11.830349 29778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:11.830404 29778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:11.841018 29778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:11.842422 29778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:11.843797 29778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:11.856848 29783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:11.856791 29784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:11.856827 29786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:11.857472 29778 server_base.cc:1034] running on GCE node
I20260620 07:08:11.858297 29778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:11.859918 29778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:11.861176 29778 hybrid_clock.cc:648] HybridClock initialized: now 1781939291861118 us; error 47 us; skew 500 ppm
I20260620 07:08:11.861549 29778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:11 dist-test-slave-wvz3 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:11.864782 29778 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:11.866376 29778 webserver.cc:460] Webserver started at http://127.11.81.254:39981/ using document root <none> and password file <none>
I20260620 07:08:11.867087 29778 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:11.867206 29778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:11.867540 29778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:11.870684 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "c5ff6e847fe84c83ba6a3f1e125189f4"
format_stamp: "Formatted at 2026-06-20 07:08:11 on dist-test-slave-wvz3"
server_key: "30cf4c8f19a623b404167adff0e26df9"
server_key_iv: "a4fd0bf8da516a8d98b7d24a8a4d79a4"
server_key_version: "encryptionkey@0"
I20260620 07:08:11.871637 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "c5ff6e847fe84c83ba6a3f1e125189f4"
format_stamp: "Formatted at 2026-06-20 07:08:11 on dist-test-slave-wvz3"
server_key: "30cf4c8f19a623b404167adff0e26df9"
server_key_iv: "a4fd0bf8da516a8d98b7d24a8a4d79a4"
server_key_version: "encryptionkey@0"
I20260620 07:08:11.877857 29778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 07:08:11.881624 29793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:11.883178 29778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:08:11.883399 29778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c5ff6e847fe84c83ba6a3f1e125189f4"
format_stamp: "Formatted at 2026-06-20 07:08:11 on dist-test-slave-wvz3"
server_key: "30cf4c8f19a623b404167adff0e26df9"
server_key_iv: "a4fd0bf8da516a8d98b7d24a8a4d79a4"
server_key_version: "encryptionkey@0"
I20260620 07:08:11.883607 29778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:11.900811 29778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:11.905239 29778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:11.905527 29778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:11.923853 29778 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:44971
I20260620 07:08:11.923877 29844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:44971 every 8 connection(s)
I20260620 07:08:11.925269 29778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:11.926930 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29778
I20260620 07:08:11.927109 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:11.927640 11591 external_mini_cluster.cc:1428] Setting key 1ae566a5338c099e2e3c50f5dac847d3
I20260620 07:08:11.929877 29845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:08:11 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:11.943080 29845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: Bootstrap starting.
I20260620 07:08:11.947487 29845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:11.948802 29845 log.cc:824] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:11.951921 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:11.929567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.11.81.254:44971)
0620 07:08:11.930368 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:11.930380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:11.930420 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:08:11.932084 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:11.932106 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:11.932515 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:11.932847 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:11.934574 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:11.936055 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:11.937867 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:11.938560 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:11.938695 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:11.943542 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:11.943554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:11.943576 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:11.943627 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:11.947506 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:11.948389 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:11.948399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:11.948408 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:11.948646 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:11.949069 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:11.949073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:11.949075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:11.949655 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:11.949886 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:11.950412 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 07:08:11.950845 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":360,"threads_started":1}
I20260620 07:08:11.952446 29845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: No bootstrap required, opened a new log
I20260620 07:08:11.960193 29845 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } }
I20260620 07:08:11.960636 29845 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:11.960762 29845 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ff6e847fe84c83ba6a3f1e125189f4, State: Initialized, Role: FOLLOWER
I20260620 07:08:11.961457 29845 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } }
I20260620 07:08:11.961680 29845 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:11.961778 29845 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:11.961907 29845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:11.964082 29845 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } }
I20260620 07:08:11.964689 29845 leader_election.cc:302] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ff6e847fe84c83ba6a3f1e125189f4; no voters:
I20260620 07:08:11.965305 29845 leader_election.cc:288] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:11.965471 29850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:11.966089 29850 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [term 1 LEADER]: Becoming Leader. State: Replica: c5ff6e847fe84c83ba6a3f1e125189f4, State: Running, Role: LEADER
I20260620 07:08:11.966640 29850 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } }
I20260620 07:08:11.967252 29845 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:11.969542 29852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ff6e847fe84c83ba6a3f1e125189f4. Latest consensus state: current_term: 1 leader_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } } }
I20260620 07:08:11.969529 29851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ff6e847fe84c83ba6a3f1e125189f4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 44971 } } }
I20260620 07:08:11.969965 29852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:11.969985 29851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:11.971390 29855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:11.977279 29855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:11.987020 29855 catalog_manager.cc:1347] Generated new cluster ID: 3a8e4d192c4c4b888165c5b2c23d9600
I20260620 07:08:11.987156 29855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:12.009402 29855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:12.011065 29855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:12.027140 29855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: Generated new TSK 0
I20260620 07:08:12.028313 29855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:08:12.086582 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:12.427026 29871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:12.427371 29871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:12.427534 29871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:12.435035 29871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:12.435180 29871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:12.435361 29871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:12.445703 29871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:12.447065 29871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:12.448436 29871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:12.462128 29876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:12.462095 29877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:12.462174 29879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:12.463012 29871 server_base.cc:1034] running on GCE node
I20260620 07:08:12.463706 29871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:12.464712 29871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:12.465919 29871 hybrid_clock.cc:648] HybridClock initialized: now 1781939292465862 us; error 52 us; skew 500 ppm
I20260620 07:08:12.466259 29871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:12 dist-test-slave-wvz3 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:12.469631 29871 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:08:12.471493 29871 webserver.cc:460] Webserver started at http://127.11.81.193:42627/ using document root <none> and password file <none>
I20260620 07:08:12.472211 29871 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:12.472334 29871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:12.472684 29871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:12.475889 29871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "b4924475067a4fa79b244bc290a5e71f"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "a25cb3641cdb16e618adc20ce6c4f10c"
server_key_iv: "4aad3e93175182fe7b3d626c82790aa3"
server_key_version: "encryptionkey@0"
I20260620 07:08:12.476696 29871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "b4924475067a4fa79b244bc290a5e71f"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "a25cb3641cdb16e618adc20ce6c4f10c"
server_key_iv: "4aad3e93175182fe7b3d626c82790aa3"
server_key_version: "encryptionkey@0"
I20260620 07:08:12.482317 29871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:08:12.485954 29886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:12.487411 29871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:12.487563 29871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "b4924475067a4fa79b244bc290a5e71f"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "a25cb3641cdb16e618adc20ce6c4f10c"
server_key_iv: "4aad3e93175182fe7b3d626c82790aa3"
server_key_version: "encryptionkey@0"
I20260620 07:08:12.487749 29871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:12.509327 29871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:12.513900 29871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:12.514204 29871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:12.515313 29871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:12.516808 29871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:12.516891 29871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:12.517005 29871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:12.517057 29871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:12.549055 29871 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:45351
I20260620 07:08:12.549105 29998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:45351 every 8 connection(s)
I20260620 07:08:12.550405 29871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:12.550926 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 29871
I20260620 07:08:12.551096 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:12.551712 11591 external_mini_cluster.cc:1428] Setting key 8876994e36f13ccc3287e826cceedb26
Jun 20 07:08:12 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:12.572448 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:12.553159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:39601 (local address 127.11.81.254:44971)
0620 07:08:12.553616 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:12.553622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:12.555013 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 07:08:12.556930 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:12.556936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:12.557017 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:12.557126 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:12.558710 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:12.559761 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:12.561404 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:12.561959 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:12.562090 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:12.566378 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:12.566383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:12.566387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:12.566425 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:12.569183 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:12.570056 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:12.570061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:12.570063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:12.570184 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:12.570633 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:12.570636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:12.570638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:12.570911 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:12.571020 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:12.571972 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 07:08:12.572156 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260620 07:08:12.573895 30002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:12.553838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44971 (local address 127.11.81.193:39601)
0620 07:08:12.554830 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:12.554888 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:12.556639 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:12.557342 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:12.557350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:12.557941 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:12.558536 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:12.558556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:12.559955 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:12.559959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:12.561257 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:12.561271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:12.562231 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:12.562238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:12.562546 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:12.563334 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:12.563365 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:12.566096 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:12.569343 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:12.569352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:12.569372 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:12.569953 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:12.570282 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:12.570287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:12.570290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:12.570535 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:12.571018 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:12.571026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:12.571530 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 07:08:12.571846 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 07:08:12.572411 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":311,"threads_started":1}
I20260620 07:08:12.576562 29999 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44971
I20260620 07:08:12.576938 29999 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:12.577705 29999 heartbeater.cc:511] Master 127.11.81.254:44971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
I20260620 07:08:12.580215 29810 ts_manager.cc:194] Registered new tserver with Master: b4924475067a4fa79b244bc290a5e71f (127.11.81.193:45351)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
I20260620 07:08:12.581830 29810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:39601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:08:12.611120 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:12.943645 30005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:12.943961 30005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:12.944061 30005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:12.951435 30005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:12.951560 30005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:12.951690 30005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:12.961752 30005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:12.963024 30005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:12.964421 30005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:12.977856 30013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:12.977929 30011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:12.977893 30010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:12.978852 30005 server_base.cc:1034] running on GCE node
I20260620 07:08:12.979548 30005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:12.980573 30005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:12.981838 30005 hybrid_clock.cc:648] HybridClock initialized: now 1781939292981774 us; error 51 us; skew 500 ppm
I20260620 07:08:12.982172 30005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:12 dist-test-slave-wvz3 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:12.985577 30005 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:08:12.987187 30005 webserver.cc:460] Webserver started at http://127.11.81.194:44169/ using document root <none> and password file <none>
I20260620 07:08:12.987953 30005 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:12.988075 30005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:12.988375 30005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:12.991570 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "9c37011617be4169ace33134ff59ebf2"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "e7056f5c5424857ea0feb7893efd2619"
server_key_iv: "81f68c10661cc52793abee0c3e28ddeb"
server_key_version: "encryptionkey@0"
I20260620 07:08:12.992440 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "9c37011617be4169ace33134ff59ebf2"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "e7056f5c5424857ea0feb7893efd2619"
server_key_iv: "81f68c10661cc52793abee0c3e28ddeb"
server_key_version: "encryptionkey@0"
I20260620 07:08:12.997886 30005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:13.002276 30020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.004122 30005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:08:13.004324 30005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "9c37011617be4169ace33134ff59ebf2"
format_stamp: "Formatted at 2026-06-20 07:08:12 on dist-test-slave-wvz3"
server_key: "e7056f5c5424857ea0feb7893efd2619"
server_key_iv: "81f68c10661cc52793abee0c3e28ddeb"
server_key_version: "encryptionkey@0"
I20260620 07:08:13.004560 30005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:13.046131 30005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:13.050841 30005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:13.051143 30005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:13.052251 30005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:13.053762 30005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:13.053853 30005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.053972 30005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:13.054024 30005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.084741 30005 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:35129
I20260620 07:08:13.084806 30132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:35129 every 8 connection(s)
I20260620 07:08:13.086038 30005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:13.086438 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30005
I20260620 07:08:13.086614 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:13.087255 11591 external_mini_cluster.cc:1428] Setting key cd2f45767e0eaf548ad49da314d70c33
Jun 20 07:08:13 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:13.108158 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.088871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:56205 (local address 127.11.81.254:44971)
0620 07:08:13.089331 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.089338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.090612 (+ 1274us) server_negotiation.cc:407] Connection header received
0620 07:08:13.092771 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.092777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.092861 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.092955 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:13.094601 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.095385 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.096996 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.097465 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.097583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.101831 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:13.101838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:13.101842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:13.101884 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:13.104633 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.105577 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.105581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.105583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.105720 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.106211 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.106214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.106216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.106519 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:13.106635 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.107610 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.107847 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260620 07:08:13.109732 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.089576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44971 (local address 127.11.81.194:56205)
0620 07:08:13.090449 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:13.090501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:13.092436 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:13.093238 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:13.093249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:13.093814 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:13.094416 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.094434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.095541 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.095545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.096820 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.096837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.097704 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.097711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.098022 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.098721 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:13.098756 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:13.101474 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:13.104818 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:13.104828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:13.104849 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:13.105468 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:13.105826 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:13.105831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:13.105835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:13.106100 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:13.106638 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:13.106647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:13.107148 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 07:08:13.107495 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 07:08:13.108113 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":360,"threads_started":1}
I20260620 07:08:13.111315 30133 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44971
I20260620 07:08:13.111649 30133 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:13.112437 30133 heartbeater.cc:511] Master 127.11.81.254:44971 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:08:13.114022 29810 ts_manager.cc:194] Registered new tserver with Master: 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129)
I20260620 07:08:13.114872 29810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:56205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:08:13.147090 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:13.476217 30139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:13.476518 30139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:13.476650 30139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:13.487406 30139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:13.487527 30139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:13.487708 30139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:13.498191 30139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:13.499629 30139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:13.500959 30139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:13.514232 30147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:13.514240 30145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:13.514828 30139 server_base.cc:1034] running on GCE node
W20260620 07:08:13.514287 30144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:13.515692 30139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:13.516810 30139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:13.518030 30139 hybrid_clock.cc:648] HybridClock initialized: now 1781939293517975 us; error 48 us; skew 500 ppm
I20260620 07:08:13.518353 30139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:13 dist-test-slave-wvz3 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:13.521611 30139 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:08:13.523154 30139 webserver.cc:460] Webserver started at http://127.11.81.195:40017/ using document root <none> and password file <none>
I20260620 07:08:13.523960 30139 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:13.524082 30139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:13.524384 30139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:13.527468 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "c9c94e18113d4094a081499264af89f0"
format_stamp: "Formatted at 2026-06-20 07:08:13 on dist-test-slave-wvz3"
server_key: "861e80fbcf6e598c67b796b145b0acb6"
server_key_iv: "37ae631e71ee33448eeedeb2759163cf"
server_key_version: "encryptionkey@0"
I20260620 07:08:13.528249 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c94e18113d4094a081499264af89f0"
format_stamp: "Formatted at 2026-06-20 07:08:13 on dist-test-slave-wvz3"
server_key: "861e80fbcf6e598c67b796b145b0acb6"
server_key_iv: "37ae631e71ee33448eeedeb2759163cf"
server_key_version: "encryptionkey@0"
I20260620 07:08:13.533612 30139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:13.537428 30154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.538987 30139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:08:13.539172 30139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "c9c94e18113d4094a081499264af89f0"
format_stamp: "Formatted at 2026-06-20 07:08:13 on dist-test-slave-wvz3"
server_key: "861e80fbcf6e598c67b796b145b0acb6"
server_key_iv: "37ae631e71ee33448eeedeb2759163cf"
server_key_version: "encryptionkey@0"
I20260620 07:08:13.539433 30139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:13.559103 30139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:13.563657 30139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:13.563934 30139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:13.564960 30139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:13.566385 30139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:13.566464 30139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.566589 30139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:13.566642 30139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:13.585376 29999 heartbeater.cc:503] Master 127.11.81.254:44971 was elected leader, sending a full tablet report...
I20260620 07:08:13.600983 30139 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:32783
I20260620 07:08:13.601027 30266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:32783 every 8 connection(s)
I20260620 07:08:13.602284 30139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:13.611580 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30139
I20260620 07:08:13.611752 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:13.612154 11591 external_mini_cluster.cc:1428] Setting key ac34aad1e54473a64d9dbc9b6f9a869c
Jun 20 07:08:13 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:13.622627 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.604950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:46365 (local address 127.11.81.254:44971)
0620 07:08:13.605132 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.605137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.606382 (+ 1245us) server_negotiation.cc:407] Connection header received
0620 07:08:13.608299 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.608305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.608389 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.608477 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:13.610065 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.610797 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.612516 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.613024 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.613138 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.616611 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:13.616616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:13.616620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:13.616703 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:13.619367 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.620193 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.620198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.620200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.620356 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.620885 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.620890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.620892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.621227 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:13.621379 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.622139 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.622360 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:08:13.623586 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.605408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:44971 (local address 127.11.81.195:46365)
0620 07:08:13.606201 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:13.606239 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:13.608004 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:13.608747 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:13.608763 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:13.609297 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:13.609886 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.609904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.610954 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.610958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.612372 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.612386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.613253 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.613261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.613485 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.614097 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:13.614125 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:13.616387 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:13.619519 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:13.619526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:13.619543 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:13.620025 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:13.620501 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:13.620506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:13.620508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:13.620714 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:13.621356 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:13.621363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:13.621727 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 07:08:13.622005 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 07:08:13.622428 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260620 07:08:13.625221 30267 heartbeater.cc:347] Connected to a master server at 127.11.81.254:44971
I20260620 07:08:13.625588 30267 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:13.626467 30267 heartbeater.cc:511] Master 127.11.81.254:44971 requested a full tablet report, sending...
I20260620 07:08:13.628177 29810 ts_manager.cc:194] Registered new tserver with Master: c9c94e18113d4094a081499264af89f0 (127.11.81.195:32783)
I20260620 07:08:13.629029 29810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:46365
I20260620 07:08:13.639969 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:13.657320 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.644218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.11.81.254:44971)
0620 07:08:13.644465 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.644471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.644845 (+ 374us) server_negotiation.cc:407] Connection header received
0620 07:08:13.644915 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.644919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.644994 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.645078 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:13.646448 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.647190 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.648629 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.649062 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.649170 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.651955 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:13.651960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:13.651964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:13.652002 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:13.654595 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.655424 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.655428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.655431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.655579 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:13.656004 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:13.656007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:13.656009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:13.656322 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:13.656537 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.656834 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.656983 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:08:13.661795 29810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:13.668918 29810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:13.695304 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.683371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:32783 (local address 127.0.0.1:35624)
0620 07:08:13.686242 (+ 2871us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:13.686287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:13.686486 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:13.687576 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:13.687584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:13.687615 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:13.688222 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.688244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.690269 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.690275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.692670 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.692681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.694142 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.694149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.694385 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.694433 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:08:13.694581 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:08:13.694796 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"mutex_wait_us":116,"thread_start_us":2374,"threads_started":1}
I20260620 07:08:13.695995 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.685445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.11.81.194:35129)
0620 07:08:13.686344 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.686355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.687044 (+ 689us) server_negotiation.cc:407] Connection header received
0620 07:08:13.687364 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.687372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.687618 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.687779 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:13.688625 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.689648 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.692830 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.694392 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.694536 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.694727 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.694960 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.695119 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":565,"threads_started":1}
I20260620 07:08:13.696184 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.683643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.11.81.195:32783)
0620 07:08:13.684293 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.684300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.686452 (+ 2152us) server_negotiation.cc:407] Connection header received
0620 07:08:13.686793 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.686800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.687068 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.687241 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:13.688448 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.690115 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.692933 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.693925 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.694058 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.694328 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.695626 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.695833 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":482,"threads_started":1}
I20260620 07:08:13.696632 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.685104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35129 (local address 127.0.0.1:50212)
0620 07:08:13.686903 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:13.686919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:13.687147 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:13.687962 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:13.687966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:13.687990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:13.688429 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.688440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.689789 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.689794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.692601 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.692622 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.694614 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.694620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.694787 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.694818 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:08:13.694912 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:08:13.695048 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"mutex_wait_us":153,"thread_start_us":1468,"threads_started":1}
I20260620 07:08:13.696837 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.682823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38156)
0620 07:08:13.686377 (+ 3554us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:13.686392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:13.686585 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:13.687622 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:13.687625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:13.687643 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:13.688277 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.688294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.690345 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.690350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.692603 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:13.692621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.695609 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.695616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:13.695831 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.695864 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:08:13.695949 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:08:13.696182 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2303,"mutex_wait_us":82,"thread_start_us":2173,"threads_started":1}
I20260620 07:08:13.697337 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:13.682804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.11.81.193:45351)
0620 07:08:13.683824 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:13.683844 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:13.686487 (+ 2643us) server_negotiation.cc:407] Connection header received
0620 07:08:13.686783 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:13.686791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:13.687068 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:13.687236 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:13.688506 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.690136 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.692870 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:13.694386 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:13.694544 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:13.694756 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:13.696679 (+ 1923us) server_negotiation.cc:299] Negotiation successful
0620 07:08:13.697033 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":1122,"threads_started":1}
I20260620 07:08:13.700023 30202 tablet_service.cc:1467] Processing CreateTablet for tablet b97209860b374d19bb9114266385e414 (DEFAULT_TABLE table=test-table [id=d599f03ba73c465bbe6beff96db61c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:13.700268 30068 tablet_service.cc:1467] Processing CreateTablet for tablet b97209860b374d19bb9114266385e414 (DEFAULT_TABLE table=test-table [id=d599f03ba73c465bbe6beff96db61c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:13.700824 29934 tablet_service.cc:1467] Processing CreateTablet for tablet b97209860b374d19bb9114266385e414 (DEFAULT_TABLE table=test-table [id=d599f03ba73c465bbe6beff96db61c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:13.701501 30202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97209860b374d19bb9114266385e414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:13.701660 30068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97209860b374d19bb9114266385e414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:13.702140 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97209860b374d19bb9114266385e414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:13.713140 30285 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Bootstrap starting.
I20260620 07:08:13.717289 30285 tablet_bootstrap.cc:654] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:13.717403 30286 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Bootstrap starting.
I20260620 07:08:13.718926 30285 log.cc:824] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:13.720460 30287 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Bootstrap starting.
I20260620 07:08:13.721200 30286 tablet_bootstrap.cc:654] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:13.722669 30286 log.cc:824] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:13.724193 30287 tablet_bootstrap.cc:654] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:13.725440 30287 log.cc:824] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:13.726298 30286 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: No bootstrap required, opened a new log
I20260620 07:08:13.726598 30286 ts_tablet_manager.cc:1397] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 07:08:13.727921 30285 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: No bootstrap required, opened a new log
I20260620 07:08:13.728214 30285 ts_tablet_manager.cc:1397] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20260620 07:08:13.729391 30287 tablet_bootstrap.cc:492] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: No bootstrap required, opened a new log
I20260620 07:08:13.729622 30287 ts_tablet_manager.cc:1397] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:08:13.737201 30287 raft_consensus.cc:348] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.737622 30287 raft_consensus.cc:374] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:13.737329 30286 raft_consensus.cc:348] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.737723 30287 raft_consensus.cc:729] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c37011617be4169ace33134ff59ebf2, State: Initialized, Role: FOLLOWER
I20260620 07:08:13.737787 30286 raft_consensus.cc:374] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:13.737890 30286 raft_consensus.cc:729] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4924475067a4fa79b244bc290a5e71f, State: Initialized, Role: FOLLOWER
I20260620 07:08:13.738426 30287 consensus_queue.cc:260] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.738484 30286 consensus_queue.cc:260] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.738658 30285 raft_consensus.cc:348] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.739130 30285 raft_consensus.cc:374] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:13.739236 30285 raft_consensus.cc:729] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c94e18113d4094a081499264af89f0, State: Initialized, Role: FOLLOWER
I20260620 07:08:13.739704 30133 heartbeater.cc:503] Master 127.11.81.254:44971 was elected leader, sending a full tablet report...
I20260620 07:08:13.740024 30285 consensus_queue.cc:260] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:13.740306 30287 ts_tablet_manager.cc:1428] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 07:08:13.744302 30267 heartbeater.cc:503] Master 127.11.81.254:44971 was elected leader, sending a full tablet report...
I20260620 07:08:13.745639 30286 ts_tablet_manager.cc:1428] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Time spent starting tablet: real 0.019s user 0.019s sys 0.001s
I20260620 07:08:13.747427 30285 ts_tablet_manager.cc:1428] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260620 07:08:13.803045 30000 tablet.cc:2369] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:13.838603 30134 tablet.cc:2369] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:13.853969 30268 tablet.cc:2369] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:14.159878 30292 raft_consensus.cc:482] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:14.160171 30292 raft_consensus.cc:504] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:14.161994 30292 leader_election.cc:288] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9c94e18113d4094a081499264af89f0 (127.11.81.195:32783), 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129)
I20260620 07:08:14.164927 30293 raft_consensus.cc:482] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:14.165274 30293 raft_consensus.cc:504] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:14.168171 30293 leader_election.cc:288] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4924475067a4fa79b244bc290a5e71f (127.11.81.193:45351), 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129)
I20260620 07:08:14.171271 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.162440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:39399 (local address 127.11.81.194:35129)
0620 07:08:14.162790 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.162794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.164031 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 07:08:14.164151 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.164155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.164212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.164279 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:14.164800 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.165827 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.168607 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170015 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.170142 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.170217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.170807 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.170961 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260620 07:08:14.171590 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.162400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:51977 (local address 127.11.81.195:32783)
0620 07:08:14.162671 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.162678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.163447 (+ 769us) server_negotiation.cc:407] Connection header received
0620 07:08:14.163702 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.163708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.163790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.163892 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:14.164463 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.165375 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.169067 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170348 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.170467 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.170543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.171130 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.171325 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":123}
I20260620 07:08:14.171842 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.162623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:32783 (local address 127.11.81.193:51977)
0620 07:08:14.163318 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.163354 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:14.163518 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:14.163941 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:14.163946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:14.163976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:14.164320 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.164331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.165498 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.165503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.168903 (+ 3400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.168919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170768 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.170774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.170947 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.170979 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:08:14.171279 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 07:08:14.171473 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":360,"threads_started":1}
I20260620 07:08:14.172605 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.162630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35129 (local address 127.11.81.193:39399)
0620 07:08:14.163918 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.163935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:14.164043 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:14.164323 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:14.164327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:14.164349 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:14.164629 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.164637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.165967 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.165973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.168400 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.168415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170423 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.170429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.170613 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.170650 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:08:14.171891 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0620 07:08:14.172177 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":48,"thread_start_us":892,"threads_started":1}
I20260620 07:08:14.172986 30222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "b4924475067a4fa79b244bc290a5e71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c94e18113d4094a081499264af89f0" is_pre_election: true
I20260620 07:08:14.173568 30222 raft_consensus.cc:2454] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4924475067a4fa79b244bc290a5e71f in term 0.
I20260620 07:08:14.173661 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "b4924475067a4fa79b244bc290a5e71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c37011617be4169ace33134ff59ebf2" is_pre_election: true
I20260620 07:08:14.174230 30088 raft_consensus.cc:2454] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4924475067a4fa79b244bc290a5e71f in term 0.
I20260620 07:08:14.174382 29887 leader_election.cc:302] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4924475067a4fa79b244bc290a5e71f, c9c94e18113d4094a081499264af89f0; no voters:
I20260620 07:08:14.174854 30292 raft_consensus.cc:2781] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:14.175031 30292 raft_consensus.cc:482] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:14.175138 30292 raft_consensus.cc:3037] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:14.177029 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.167161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:35129 (local address 127.11.81.195:60045)
0620 07:08:14.168717 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.168736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:14.168857 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:14.170348 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:14.170353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:14.170379 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:14.170753 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.170775 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.172592 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.172597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.174615 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.174631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.176090 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.176096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.176287 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.176313 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:08:14.176415 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:08:14.176592 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":5,"thread_start_us":1288,"threads_started":1}
I20260620 07:08:14.177089 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.167642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:60045 (local address 127.11.81.194:35129)
0620 07:08:14.170016 (+ 2374us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.170022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.170044 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:08:14.170102 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.170107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.170189 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.170918 (+ 729us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:14.171193 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.172370 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.174838 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.175937 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.176102 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.176178 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.176620 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.176765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230,"thread_start_us":3085,"threads_started":1}
I20260620 07:08:14.177302 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.167538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:52245 (local address 127.11.81.193:45351)
0620 07:08:14.168012 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.168018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.168038 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:08:14.168098 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.168102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.168172 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.168272 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:14.169564 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170648 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.174983 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.176277 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.176449 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.176522 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.176922 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.177070 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260620 07:08:14.177873 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.167161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.11.81.195:52245)
0620 07:08:14.167818 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.167839 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:14.167974 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:14.168987 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:14.168992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:14.169017 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:14.169421 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.169433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.170900 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.170906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.174775 (+ 3869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:14.174790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.176502 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.176509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:14.176700 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.176737 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:08:14.177458 (+ 721us) client_negotiation.cc:240] Negotiation successful
0620 07:08:14.177612 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":409,"threads_started":1}
I20260620 07:08:14.177953 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "c9c94e18113d4094a081499264af89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c37011617be4169ace33134ff59ebf2" is_pre_election: true
I20260620 07:08:14.178198 30088 raft_consensus.cc:2454] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c94e18113d4094a081499264af89f0 in term 0.
I20260620 07:08:14.178673 30157 leader_election.cc:302] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c37011617be4169ace33134ff59ebf2, c9c94e18113d4094a081499264af89f0; no voters:
I20260620 07:08:14.178785 29954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "c9c94e18113d4094a081499264af89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4924475067a4fa79b244bc290a5e71f" is_pre_election: true
I20260620 07:08:14.179143 30293 raft_consensus.cc:2781] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:14.179308 30293 raft_consensus.cc:482] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:14.179200 30292 raft_consensus.cc:504] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:14.179404 30293 raft_consensus.cc:3037] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:14.179682 29954 raft_consensus.cc:2379] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9c94e18113d4094a081499264af89f0 in current term 1: Already voted for candidate b4924475067a4fa79b244bc290a5e71f in this term.
I20260620 07:08:14.180447 30222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "b4924475067a4fa79b244bc290a5e71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c94e18113d4094a081499264af89f0"
I20260620 07:08:14.180552 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "b4924475067a4fa79b244bc290a5e71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c37011617be4169ace33134ff59ebf2"
I20260620 07:08:14.180708 30088 raft_consensus.cc:3037] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:14.181496 30292 leader_election.cc:288] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [CANDIDATE]: Term 1 election: Requested vote from peers c9c94e18113d4094a081499264af89f0 (127.11.81.195:32783), 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129)
I20260620 07:08:14.182271 30293 raft_consensus.cc:504] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:14.182670 30222 raft_consensus.cc:2379] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b4924475067a4fa79b244bc290a5e71f in current term 1: Already voted for candidate c9c94e18113d4094a081499264af89f0 in this term.
I20260620 07:08:14.182904 30293 leader_election.cc:288] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [CANDIDATE]: Term 1 election: Requested vote from peers b4924475067a4fa79b244bc290a5e71f (127.11.81.193:45351), 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129)
I20260620 07:08:14.183072 30088 raft_consensus.cc:2454] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4924475067a4fa79b244bc290a5e71f in term 1.
I20260620 07:08:14.183355 29954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "c9c94e18113d4094a081499264af89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4924475067a4fa79b244bc290a5e71f"
I20260620 07:08:14.183622 29954 raft_consensus.cc:2379] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9c94e18113d4094a081499264af89f0 in current term 1: Already voted for candidate b4924475067a4fa79b244bc290a5e71f in this term.
I20260620 07:08:14.183950 29889 leader_election.cc:302] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c37011617be4169ace33134ff59ebf2, b4924475067a4fa79b244bc290a5e71f; no voters:
I20260620 07:08:14.184257 30292 raft_consensus.cc:2781] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:14.184479 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97209860b374d19bb9114266385e414" candidate_uuid: "c9c94e18113d4094a081499264af89f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c37011617be4169ace33134ff59ebf2"
I20260620 07:08:14.184693 30292 raft_consensus.cc:686] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 LEADER]: Becoming Leader. State: Replica: b4924475067a4fa79b244bc290a5e71f, State: Running, Role: LEADER
I20260620 07:08:14.184818 30088 raft_consensus.cc:2379] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9c94e18113d4094a081499264af89f0 in current term 1: Already voted for candidate b4924475067a4fa79b244bc290a5e71f in this term.
I20260620 07:08:14.185214 30292 consensus_queue.cc:237] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } }
I20260620 07:08:14.185467 30157 leader_election.cc:302] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9c94e18113d4094a081499264af89f0; no voters: 9c37011617be4169ace33134ff59ebf2, b4924475067a4fa79b244bc290a5e71f
I20260620 07:08:14.185770 30293 raft_consensus.cc:2726] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 07:08:14.190013 29810 catalog_manager.cc:5495] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f reported cstate change: term changed from 0 to 1, leader changed from <none> to b4924475067a4fa79b244bc290a5e71f (127.11.81.193). New cstate: current_term: 1 leader_uuid: "b4924475067a4fa79b244bc290a5e71f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4924475067a4fa79b244bc290a5e71f" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 45351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:14.266310 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.259674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.11.81.193:45351)
0620 07:08:14.259862 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.259867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.260290 (+ 423us) server_negotiation.cc:407] Connection header received
0620 07:08:14.260447 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.260452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.260547 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.260681 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:14.261242 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.262030 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.263755 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.264205 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.264313 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.264952 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:14.265530 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:14.265687 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.265985 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.266035 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:08:14.274752 30088 raft_consensus.cc:1261] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 1 FOLLOWER]: Refusing update from remote peer b4924475067a4fa79b244bc290a5e71f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:14.274768 30222 raft_consensus.cc:1261] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Refusing update from remote peer b4924475067a4fa79b244bc290a5e71f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:14.275635 30299 consensus_queue.cc:1035] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c94e18113d4094a081499264af89f0" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:14.275976 30292 consensus_queue.cc:1035] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c37011617be4169ace33134ff59ebf2" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:14.303687 29810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:14.303957 29810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:14.307718 29810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5ff6e847fe84c83ba6a3f1e125189f4: Sending DeleteTablet for 3 replicas of tablet b97209860b374d19bb9114266385e414
I20260620 07:08:14.309015 29934 tablet_service.cc:1514] Processing DeleteTablet for tablet b97209860b374d19bb9114266385e414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:14 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38156
I20260620 07:08:14.309010 30202 tablet_service.cc:1514] Processing DeleteTablet for tablet b97209860b374d19bb9114266385e414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:14 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:35624
I20260620 07:08:14.309130 30068 tablet_service.cc:1514] Processing DeleteTablet for tablet b97209860b374d19bb9114266385e414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:14 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:50212
I20260620 07:08:14.311190 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29871
I20260620 07:08:14.321633 30317 tablet_replica.cc:330] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: stopping tablet replica
I20260620 07:08:14.322270 30318 tablet_replica.cc:330] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: stopping tablet replica
I20260620 07:08:14.322685 30318 raft_consensus.cc:2229] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:14.323179 30318 raft_consensus.cc:2258] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:14.323961 30317 raft_consensus.cc:2229] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:14.324442 30317 raft_consensus.cc:2258] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:14.326361 30317 ts_tablet_manager.cc:1905] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:14.326363 30318 ts_tablet_manager.cc:1905] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:14.330962 30319 tablet_replica.cc:330] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: stopping tablet replica
I20260620 07:08:14.331540 30319 raft_consensus.cc:2229] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:14.332350 30319 raft_consensus.cc:2258] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:14.333616 30318 ts_tablet_manager.cc:1918] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:14.333807 30318 log.cc:1196] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/b97209860b374d19bb9114266385e414
I20260620 07:08:14.333779 30317 ts_tablet_manager.cc:1918] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:14.334059 30317 log.cc:1196] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/b97209860b374d19bb9114266385e414
I20260620 07:08:14.334316 30318 ts_tablet_manager.cc:1939] T b97209860b374d19bb9114266385e414 P 9c37011617be4169ace33134ff59ebf2: Deleting consensus metadata
I20260620 07:08:14.334649 30317 ts_tablet_manager.cc:1939] T b97209860b374d19bb9114266385e414 P c9c94e18113d4094a081499264af89f0: Deleting consensus metadata
I20260620 07:08:14.336357 29796 catalog_manager.cc:4841] TS 9c37011617be4169ace33134ff59ebf2 (127.11.81.194:35129): tablet b97209860b374d19bb9114266385e414 (table test-table [id=d599f03ba73c465bbe6beff96db61c94]) successfully deleted
I20260620 07:08:14.336833 29794 catalog_manager.cc:4841] TS c9c94e18113d4094a081499264af89f0 (127.11.81.195:32783): tablet b97209860b374d19bb9114266385e414 (table test-table [id=d599f03ba73c465bbe6beff96db61c94]) successfully deleted
Jun 20 07:08:14 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:08:14.339177 30319 ts_tablet_manager.cc:1905] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:14.345909 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.313073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.11.81.193:45351)
0620 07:08:14.313288 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.313295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.324398 (+ 11103us) server_negotiation.cc:407] Connection header received
0620 07:08:14.324919 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.324927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.325042 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.325126 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:14.327142 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.328425 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.332777 (+ 4352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.333509 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.333618 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.339151 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:14.339166 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:14.339181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:14.339229 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:14.342606 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.343563 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.343571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.343574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.343751 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.344173 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.344178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.344182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.344643 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:14.344987 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.345289 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.345533 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:08:14.346683 29994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:14.447969 30319 ts_tablet_manager.cc:1918] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:14.448290 30319 log.cc:1196] T b97209860b374d19bb9114266385e414 P b4924475067a4fa79b244bc290a5e71f: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/b97209860b374d19bb9114266385e414
I20260620 07:08:14.448611 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29871
I20260620 07:08:14.463182 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30005
W20260620 07:08:14.463565 29797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:45351 (error 108)
I20260620 07:08:14.465816 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.465082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38168)
0620 07:08:14.465524 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.465633 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":232,"threads_started":1}
W20260620 07:08:14.466054 29797 catalog_manager.cc:4584] TS b4924475067a4fa79b244bc290a5e71f (127.11.81.193:45351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b97209860b374d19bb9114266385e414: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Jun 20 07:08:14 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:08:14.478796 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.464189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.11.81.194:35129)
0620 07:08:14.464404 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.464409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.464431 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:08:14.464537 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.464543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.464639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.464742 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:14.466006 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.466778 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.468157 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.468565 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.468712 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.473097 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:14.473115 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:14.473132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:14.473175 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:14.476127 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.476852 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.476859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.476862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.476992 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.477382 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.477387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.477389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.477834 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:14.478083 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.478332 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.478542 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:08:14.479468 30128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:14.519851 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.519382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38178)
0620 07:08:14.519591 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.519690 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:08:14.521061 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.520664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38192)
0620 07:08:14.520820 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.520896 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:08:14.570377 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30005
I20260620 07:08:14.581629 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.581110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38196)
0620 07:08:14.581356 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.581475 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:08:14.582891 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.582443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38210)
0620 07:08:14.582611 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.582705 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:08:14.585690 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30139
Jun 20 07:08:14 dist-test-slave-wvz3 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:08:14.600471 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.586361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.11.81.195:32783)
0620 07:08:14.586552 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:14.586557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:14.586579 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:08:14.586645 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:14.586648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:14.586712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:14.586814 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:14.588383 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.589139 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.590403 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:14.590808 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:14.590904 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:14.595144 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:14.595161 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:14.595178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:14.595222 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:14.597863 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.598554 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.598561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.598564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.598700 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:14.599019 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:14.599022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:14.599024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:14.599429 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:14.599645 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:14.599936 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 07:08:14.600103 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:08:14.601142 30262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:14.663820 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.663353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38222)
0620 07:08:14.663554 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.663652 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:08:14.664954 30323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:14.664477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:45351 (local address 127.0.0.1:38238)
0620 07:08:14.664705 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:14.664803 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 07:08:14.692616 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30139
I20260620 07:08:14.706343 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 29778
I20260620 07:08:14.707525 29840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:14.818390 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 29778
2026-06-20T07:08:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4603 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T07:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:14Z Disabled control of system clock
I20260620 07:08:14.892143 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38165
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:38165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:15.231813 30330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:15.232131 30330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:15.232224 30330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:15.239586 30330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:15.239710 30330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:15.239748 30330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:15.239795 30330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:15.239845 30330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:15.249349 30330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:38165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:15.250655 30330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:15.252534 30330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:15.264464 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:15.264530 30338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:15.264551 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:15.265401 30330 server_base.cc:1034] running on GCE node
I20260620 07:08:15.266244 30330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:15.267890 30330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:15.269101 30330 hybrid_clock.cc:648] HybridClock initialized: now 1781939295269042 us; error 57 us; skew 500 ppm
I20260620 07:08:15.269459 30330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:15.276918 30330 webserver.cc:460] Webserver started at http://127.11.81.254:40011/ using document root <none> and password file <none>
I20260620 07:08:15.277694 30330 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:15.277814 30330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:15.278132 30330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:15.281220 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "15e4c73392eb4d78bf6ad569bdd167b9"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "9c5e2734e2c66384b91484ca39d87948"
server_key_iv: "fe8ca3deb8b08ce90cfdc576223ab487"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.282047 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "15e4c73392eb4d78bf6ad569bdd167b9"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "9c5e2734e2c66384b91484ca39d87948"
server_key_iv: "fe8ca3deb8b08ce90cfdc576223ab487"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.287569 30330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 07:08:15.291338 30344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:15.293025 30330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:15.293231 30330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "15e4c73392eb4d78bf6ad569bdd167b9"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "9c5e2734e2c66384b91484ca39d87948"
server_key_iv: "fe8ca3deb8b08ce90cfdc576223ab487"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.293466 30330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:15.318182 30330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:15.319114 30330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:15.319427 30330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:15.337587 30330 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:38165
I20260620 07:08:15.337606 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:38165 every 8 connection(s)
I20260620 07:08:15.338948 30330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:15.343257 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:15.346830 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30330
I20260620 07:08:15.347014 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:15.347527 11591 external_mini_cluster.cc:1428] Setting key b6740d1ec8ec49ae933eaee013f25362
I20260620 07:08:15.357632 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: Bootstrap starting.
I20260620 07:08:15.361548 30396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:15.362910 30396 log.cc:824] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:15.363909 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:15.349691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.11.81.254:38165)
0620 07:08:15.350805 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:15.350831 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:15.350884 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:08:15.352771 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:15.352805 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:15.353520 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:15.354044 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:15.354639 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.356721 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.358311 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.359414 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.359617 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:15.360805 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:15.360815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:15.360834 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:15.361581 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:15.361713 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:15.361751 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:15.361858 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:15.362128 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:08:15.362478 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":346,"threads_started":1}
I20260620 07:08:15.366561 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: No bootstrap required, opened a new log
I20260620 07:08:15.374818 30396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } }
I20260620 07:08:15.375231 30396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:15.375370 30396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e4c73392eb4d78bf6ad569bdd167b9, State: Initialized, Role: FOLLOWER
I20260620 07:08:15.375955 30396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } }
I20260620 07:08:15.376169 30396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:15.376291 30396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:15.376453 30396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:15.378638 30396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } }
I20260620 07:08:15.379201 30396 leader_election.cc:302] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15e4c73392eb4d78bf6ad569bdd167b9; no voters:
I20260620 07:08:15.379829 30396 leader_election.cc:288] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:15.379987 30401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:15.381138 30401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [term 1 LEADER]: Becoming Leader. State: Replica: 15e4c73392eb4d78bf6ad569bdd167b9, State: Running, Role: LEADER
I20260620 07:08:15.381696 30401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } }
I20260620 07:08:15.382053 30396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:15.384698 30403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15e4c73392eb4d78bf6ad569bdd167b9. Latest consensus state: current_term: 1 leader_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } } }
I20260620 07:08:15.385135 30403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:15.385628 30402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15e4c73392eb4d78bf6ad569bdd167b9" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 38165 } } }
I20260620 07:08:15.385954 30402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:15.386224 30410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:15.392042 30410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:15.401309 30410 catalog_manager.cc:1347] Generated new cluster ID: 5bdced0e811f42b8ba5a0826ac2ecda9
I20260620 07:08:15.401445 30410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:15.412575 30410 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:15.414110 30410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:15.424853 30410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: Generated new TSK 0
I20260620 07:08:15.426019 30410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:15.434839 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--builtin_ntp_servers=127.11.81.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:15.772778 30420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:15.773061 30420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:15.773175 30420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:15.780717 30420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:15.780840 30420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:15.780977 30420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:15.791190 30420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:15.792610 30420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:15.794502 30420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:15.808180 30425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:15.808152 30428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:15.808115 30426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:15.809432 30420 server_base.cc:1034] running on GCE node
I20260620 07:08:15.810174 30420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:15.811317 30420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:15.812592 30420 hybrid_clock.cc:648] HybridClock initialized: now 1781939295812536 us; error 44 us; skew 500 ppm
I20260620 07:08:15.813006 30420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:15.815546 30420 webserver.cc:460] Webserver started at http://127.11.81.193:39385/ using document root <none> and password file <none>
I20260620 07:08:15.816277 30420 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:15.816393 30420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:15.816704 30420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:15.819847 30420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "c632caf2fbe04c1daf6b0673bfa27d56"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "f14d65a6b872c267e1461a933c9b7c3d"
server_key_iv: "7dea0ecdb85ef4b6c92c6583ab0ccb8c"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.820693 30420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "c632caf2fbe04c1daf6b0673bfa27d56"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "f14d65a6b872c267e1461a933c9b7c3d"
server_key_iv: "7dea0ecdb85ef4b6c92c6583ab0ccb8c"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.826449 30420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:15.830281 30434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:15.831892 30420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:15.832067 30420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "c632caf2fbe04c1daf6b0673bfa27d56"
format_stamp: "Formatted at 2026-06-20 07:08:15 on dist-test-slave-wvz3"
server_key: "f14d65a6b872c267e1461a933c9b7c3d"
server_key_iv: "7dea0ecdb85ef4b6c92c6583ab0ccb8c"
server_key_version: "encryptionkey@0"
I20260620 07:08:15.832283 30420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:15.855407 30420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:15.856314 30420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:15.856600 30420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:15.857645 30420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:15.859208 30420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:15.859320 30420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:15.859431 30420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:15.859510 30420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:15.891004 30420 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:33787
I20260620 07:08:15.891072 30546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:33787 every 8 connection(s)
I20260620 07:08:15.892341 30420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:15.899896 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30420
I20260620 07:08:15.900058 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:15.900468 11591 external_mini_cluster.cc:1428] Setting key db674f8c9258e84dcb6c30b916b15617
I20260620 07:08:15.903146 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--builtin_ntp_servers=127.11.81.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:15.905344 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:15.894971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:50825 (local address 127.11.81.254:38165)
0620 07:08:15.895443 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:15.895447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:15.896732 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 07:08:15.898569 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:15.898574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:15.898667 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:15.898766 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:15.899925 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.900756 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.902560 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.902978 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.903089 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:15.904520 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:15.904525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:15.904528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:15.904547 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:15.904589 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:15.904601 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:15.904684 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:15.904902 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:08:15.905006 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":217,"threads_started":1}
I20260620 07:08:15.906395 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:15.895629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38165 (local address 127.11.81.193:50825)
0620 07:08:15.896533 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:15.896577 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:15.898298 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:15.899011 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:15.899020 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:15.899118 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:15.899758 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:15.899776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.900910 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.900916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:15.902424 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:15.902438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:15.903209 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:15.903218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:15.903523 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:15.904201 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:15.904203 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:15.904247 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:15.904337 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:15.904688 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:15.904700 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:15.904709 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:08:15.904863 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:08:15.905306 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260620 07:08:15.908082 30547 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38165
I20260620 07:08:15.908416 30547 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:15.909200 30547 heartbeater.cc:511] Master 127.11.81.254:38165 requested a full tablet report, sending...
I20260620 07:08:15.911556 30361 ts_manager.cc:194] Registered new tserver with Master: c632caf2fbe04c1daf6b0673bfa27d56 (127.11.81.193:33787)
I20260620 07:08:15.913115 30361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.193:50825
W20260620 07:08:16.236179 30551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:16.236488 30551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:16.236627 30551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:16.244488 30551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:16.244602 30551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:16.244738 30551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:16.254047 30551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:16.255426 30551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:16.257238 30551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:16.270601 30559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:16.270651 30556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:16.270695 30557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:16.277240 30551 server_base.cc:1034] running on GCE node
I20260620 07:08:16.278000 30551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:16.279110 30551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:16.280355 30551 hybrid_clock.cc:648] HybridClock initialized: now 1781939296280307 us; error 73 us; skew 500 ppm
I20260620 07:08:16.280735 30551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:16.283759 30551 webserver.cc:460] Webserver started at http://127.11.81.194:42675/ using document root <none> and password file <none>
I20260620 07:08:16.284543 30551 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:16.284654 30551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:16.285006 30551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:16.288564 30551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "be60e28ff4d9471e946e0d5a03dc3d1a"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "92ea04a6449691a721918717d31cf709"
server_key_iv: "b6a3761cb2f399697260687fcb3dfcf0"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.289422 30551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "be60e28ff4d9471e946e0d5a03dc3d1a"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "92ea04a6449691a721918717d31cf709"
server_key_iv: "b6a3761cb2f399697260687fcb3dfcf0"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.295120 30551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:08:16.299065 30565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.300592 30551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:16.300822 30551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "be60e28ff4d9471e946e0d5a03dc3d1a"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "92ea04a6449691a721918717d31cf709"
server_key_iv: "b6a3761cb2f399697260687fcb3dfcf0"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.301054 30551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:16.316488 30551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:16.317389 30551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:16.317692 30551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:16.318749 30551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:16.320317 30551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:16.320420 30551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.320530 30551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:16.320597 30551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.352548 30551 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:33969
I20260620 07:08:16.352566 30677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:33969 every 8 connection(s)
I20260620 07:08:16.353858 30551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:16.359030 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30551
I20260620 07:08:16.359198 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:16.359651 11591 external_mini_cluster.cc:1428] Setting key b8c02e8c6ebcbb8d0bbbad3df936dd23
I20260620 07:08:16.362444 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--builtin_ntp_servers=127.11.81.212:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:16.366618 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.356493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:60205 (local address 127.11.81.254:38165)
0620 07:08:16.356688 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.356693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.358175 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 07:08:16.360085 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.360090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.360173 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.360261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:16.361588 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.362246 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.363961 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.364368 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.364476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.365824 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:16.365830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:16.365835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:16.365860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:16.365895 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:16.365908 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:16.365983 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.366197 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.366310 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:08:16.367734 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.357037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38165 (local address 127.11.81.194:60205)
0620 07:08:16.358008 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:16.358051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:16.359801 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:16.360524 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:16.360532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:16.360659 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:16.361427 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.361448 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.362407 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.362413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.363824 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.363837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.364581 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.364589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.364825 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.365505 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:16.365508 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:16.365550 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:16.365633 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:16.365996 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:16.366007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:16.366017 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:08:16.366175 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 07:08:16.366591 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":317,"threads_started":1}
I20260620 07:08:16.369185 30678 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38165
I20260620 07:08:16.369477 30678 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:16.370226 30678 heartbeater.cc:511] Master 127.11.81.254:38165 requested a full tablet report, sending...
I20260620 07:08:16.371721 30361 ts_manager.cc:194] Registered new tserver with Master: be60e28ff4d9471e946e0d5a03dc3d1a (127.11.81.194:33969)
I20260620 07:08:16.372570 30361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.194:60205
W20260620 07:08:16.689775 30681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:16.690068 30681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:16.690176 30681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:16.697746 30681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:16.697856 30681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:16.697993 30681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:16.707490 30681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:38165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:16.708802 30681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:16.710546 30681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:16.723793 30689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:16.723959 30687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:16.723872 30686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:16.725144 30681 server_base.cc:1034] running on GCE node
I20260620 07:08:16.725817 30681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:16.726760 30681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:16.727964 30681 hybrid_clock.cc:648] HybridClock initialized: now 1781939296727917 us; error 38 us; skew 500 ppm
I20260620 07:08:16.728293 30681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:16.730618 30681 webserver.cc:460] Webserver started at http://127.11.81.195:41749/ using document root <none> and password file <none>
I20260620 07:08:16.731383 30681 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:16.731487 30681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:16.731792 30681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:16.734683 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "2d93c61f79954fab997872de5aa7de73"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "51ddd2c55dcf8746bea4c0cc9027244c"
server_key_iv: "4bd2a0ae9e60d726b3d049034b178f8b"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.735519 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "2d93c61f79954fab997872de5aa7de73"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "51ddd2c55dcf8746bea4c0cc9027244c"
server_key_iv: "4bd2a0ae9e60d726b3d049034b178f8b"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.740865 30681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:08:16.744498 30695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.745985 30681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:08:16.746151 30681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "2d93c61f79954fab997872de5aa7de73"
format_stamp: "Formatted at 2026-06-20 07:08:16 on dist-test-slave-wvz3"
server_key: "51ddd2c55dcf8746bea4c0cc9027244c"
server_key_iv: "4bd2a0ae9e60d726b3d049034b178f8b"
server_key_version: "encryptionkey@0"
I20260620 07:08:16.746363 30681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:16.767823 30681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:16.768703 30681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:16.769009 30681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:16.770017 30681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:16.771595 30681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:16.771700 30681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.771826 30681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:16.771898 30681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:16.802286 30681 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:33565
I20260620 07:08:16.802349 30807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:33565 every 8 connection(s)
I20260620 07:08:16.803617 30681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:16.807022 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30681
I20260620 07:08:16.807201 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:16.807765 11591 external_mini_cluster.cc:1428] Setting key 7bf7f8ef77e5ad6c948eeae6ba0d0e66
I20260620 07:08:16.817238 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.806381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:40095 (local address 127.11.81.254:38165)
0620 07:08:16.806561 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.806565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.808371 (+ 1806us) server_negotiation.cc:407] Connection header received
0620 07:08:16.810584 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.810590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.810674 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.810761 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:16.811984 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.812758 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.814335 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.814778 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.814875 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.816347 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:16.816352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:16.816355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:16.816377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:16.816412 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:16.816425 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:16.816516 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.816866 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.816988 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:08:16.818188 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.806971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:38165 (local address 127.11.81.195:40095)
0620 07:08:16.808172 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:16.808224 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:16.810244 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:16.811003 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:16.811013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:16.811113 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:16.811813 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.811833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.812911 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.812915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.814188 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.814203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.814961 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.814966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.815182 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.815967 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:16.815972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:16.816044 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:16.816134 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:16.816544 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:16.816556 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:16.816565 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:08:16.816743 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:08:16.817121 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":363,"threads_started":1}
I20260620 07:08:16.819629 30808 heartbeater.cc:347] Connected to a master server at 127.11.81.254:38165
I20260620 07:08:16.819933 30808 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:16.820696 30808 heartbeater.cc:511] Master 127.11.81.254:38165 requested a full tablet report, sending...
I20260620 07:08:16.822461 30360 ts_manager.cc:194] Registered new tserver with Master: 2d93c61f79954fab997872de5aa7de73 (127.11.81.195:33565)
I20260620 07:08:16.823705 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:16.823768 30360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.195:40095
I20260620 07:08:16.834379 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.828289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.11.81.254:38165)
0620 07:08:16.828520 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.828527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.828830 (+ 303us) server_negotiation.cc:407] Connection header received
0620 07:08:16.828979 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.828983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.829056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.829139 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:16.829595 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.830312 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.831938 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.832409 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.832508 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.833667 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:16.833671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:16.833674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:16.833694 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:16.833729 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:16.833742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:16.833859 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.834001 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.834055 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:08:16.838697 30360 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:16.845677 30360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:16.877841 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.863780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.11.81.194:33969)
0620 07:08:16.864806 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.864816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.868677 (+ 3861us) server_negotiation.cc:407] Connection header received
0620 07:08:16.868861 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.868872 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.869174 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.869393 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:16.870515 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.872264 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.875279 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.876678 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.876799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.876999 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.877332 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.877506 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":688,"threads_started":1}
I20260620 07:08:16.877837 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.863391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:33969 (local address 127.0.0.1:58920)
0620 07:08:16.868507 (+ 5116us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:16.868524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:16.868686 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:16.869673 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:16.869682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:16.869720 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:16.870289 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.870302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.872489 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.872495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.875068 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.875084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.876853 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.876859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.877061 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.877115 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:08:16.877262 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:08:16.877464 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5004,"mutex_wait_us":44,"thread_start_us":4828,"threads_started":1}
I20260620 07:08:16.878405 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.868942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.11.81.195:33565)
0620 07:08:16.869521 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.869529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.870263 (+ 734us) server_negotiation.cc:407] Connection header received
0620 07:08:16.870562 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.870569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.870816 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.870986 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:16.871810 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.872806 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.875123 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.876602 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.876751 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.877036 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.877518 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.877674 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":297,"threads_started":1}
I20260620 07:08:16.879168 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.869568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33565 (local address 127.0.0.1:51034)
0620 07:08:16.870142 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:16.870161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:16.870367 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:16.871279 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:16.871306 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:16.871332 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:16.871626 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.871633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.872940 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.872945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.874872 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.874886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.877149 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.877153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.877329 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.877361 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:08:16.878667 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0620 07:08:16.878828 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":3,"thread_start_us":288,"threads_started":1}
I20260620 07:08:16.882735 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.862915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45888)
0620 07:08:16.867678 (+ 4763us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:16.867728 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:16.867962 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:16.876009 (+ 8047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:16.876015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:16.876051 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:16.876390 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.876403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.878075 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.878080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.880061 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:16.880075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.881940 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.881947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:16.882136 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.882188 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 07:08:16.882268 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:08:16.882410 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":779,"threads_started":1}
I20260620 07:08:16.882438 30743 tablet_service.cc:1467] Processing CreateTablet for tablet 819edf66218a476fbd60c2b2d909f229 (DEFAULT_TABLE table=test-table [id=23e427217dd64e64baffa130f99f75ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:16.882606 30613 tablet_service.cc:1467] Processing CreateTablet for tablet 819edf66218a476fbd60c2b2d909f229 (DEFAULT_TABLE table=test-table [id=23e427217dd64e64baffa130f99f75ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:16.883868 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:16.862968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.11.81.193:33787)
0620 07:08:16.875500 (+ 12532us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:16.875508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:16.875533 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:08:16.875612 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:16.875617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:16.875821 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:16.876001 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:16.876556 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.877912 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.880318 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:16.881720 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:16.883009 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:16.883306 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:16.883382 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 07:08:16.883558 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12369,"thread_start_us":13232,"threads_started":1}
I20260620 07:08:16.884195 30743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819edf66218a476fbd60c2b2d909f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:16.884426 30613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819edf66218a476fbd60c2b2d909f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:16.893473 30482 tablet_service.cc:1467] Processing CreateTablet for tablet 819edf66218a476fbd60c2b2d909f229 (DEFAULT_TABLE table=test-table [id=23e427217dd64e64baffa130f99f75ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:16.895026 30482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819edf66218a476fbd60c2b2d909f229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:16.903646 30826 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Bootstrap starting.
I20260620 07:08:16.903906 30827 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Bootstrap starting.
I20260620 07:08:16.907814 30827 tablet_bootstrap.cc:654] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:16.908401 30826 tablet_bootstrap.cc:654] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:16.908942 30827 log.cc:824] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:16.909614 30826 log.cc:824] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:16.911788 30827 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: No bootstrap required, opened a new log
I20260620 07:08:16.912040 30827 ts_tablet_manager.cc:1397] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 07:08:16.913278 30826 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: No bootstrap required, opened a new log
I20260620 07:08:16.913522 30826 ts_tablet_manager.cc:1397] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 07:08:16.916942 30547 heartbeater.cc:503] Master 127.11.81.254:38165 was elected leader, sending a full tablet report...
I20260620 07:08:16.919346 30829 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Bootstrap starting.
I20260620 07:08:16.920002 30827 raft_consensus.cc:348] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.920663 30827 raft_consensus.cc:374] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:16.920840 30827 raft_consensus.cc:729] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be60e28ff4d9471e946e0d5a03dc3d1a, State: Initialized, Role: FOLLOWER
I20260620 07:08:16.921692 30827 consensus_queue.cc:260] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.922168 30826 raft_consensus.cc:348] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.922700 30826 raft_consensus.cc:374] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:16.922820 30826 raft_consensus.cc:729] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d93c61f79954fab997872de5aa7de73, State: Initialized, Role: FOLLOWER
I20260620 07:08:16.922844 30678 heartbeater.cc:503] Master 127.11.81.254:38165 was elected leader, sending a full tablet report...
I20260620 07:08:16.923532 30827 ts_tablet_manager.cc:1428] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 07:08:16.923487 30826 consensus_queue.cc:260] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.924264 30829 tablet_bootstrap.cc:654] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:16.924816 30808 heartbeater.cc:503] Master 127.11.81.254:38165 was elected leader, sending a full tablet report...
I20260620 07:08:16.925565 30826 ts_tablet_manager.cc:1428] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
I20260620 07:08:16.925700 30829 log.cc:824] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:16.928856 30829 tablet_bootstrap.cc:492] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: No bootstrap required, opened a new log
I20260620 07:08:16.929116 30829 ts_tablet_manager.cc:1397] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 07:08:16.936538 30829 raft_consensus.cc:348] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.936941 30829 raft_consensus.cc:374] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:16.937044 30829 raft_consensus.cc:729] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c632caf2fbe04c1daf6b0673bfa27d56, State: Initialized, Role: FOLLOWER
I20260620 07:08:16.937604 30829 consensus_queue.cc:260] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:16.939316 30829 ts_tablet_manager.cc:1428] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260620 07:08:17.055490 30809 tablet.cc:2369] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:17.106006 30679 tablet.cc:2369] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:17.139483 30831 raft_consensus.cc:482] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:17.139820 30831 raft_consensus.cc:504] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:17.141552 30831 leader_election.cc:288] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c632caf2fbe04c1daf6b0673bfa27d56 (127.11.81.193:33787), 2d93c61f79954fab997872de5aa7de73 (127.11.81.195:33565)
W20260620 07:08:17.144639 30548 tablet.cc:2369] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:17.148962 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.142109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:59883 (local address 127.11.81.195:33565)
0620 07:08:17.142313 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.142320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.142894 (+ 574us) server_negotiation.cc:407] Connection header received
0620 07:08:17.143122 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.143126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.143187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.143303 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:17.143929 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.144638 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.146604 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.147820 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.147997 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.148071 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.148590 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.148722 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:08:17.149008 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.142041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:58867 (local address 127.11.81.193:33787)
0620 07:08:17.142230 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.142236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.142874 (+ 638us) server_negotiation.cc:407] Connection header received
0620 07:08:17.143062 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.143069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.143151 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.143219 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:17.143881 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.144715 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.146671 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.147884 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.148010 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.148071 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.148606 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.148730 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:08:17.149115 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.141973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.11.81.194:58867)
0620 07:08:17.142742 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.142768 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:17.142911 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:17.143270 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:17.143274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:17.143319 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:17.143698 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:17.143707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.144881 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.144886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:17.146500 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:17.146512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.148091 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.148097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:17.148374 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.148439 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 07:08:17.148533 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:08:17.148733 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":91,"thread_start_us":401,"threads_started":1}
I20260620 07:08:17.149127 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.141973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:33565 (local address 127.11.81.194:59883)
0620 07:08:17.142776 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.142791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:17.142975 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:17.143349 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:17.143354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:17.143382 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:17.143766 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:17.143775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.144783 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.144788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:17.146414 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:17.146426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.148067 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.148077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:17.148373 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.148447 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 07:08:17.148532 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:08:17.148712 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":275,"spinlock_wait_cycles":28416,"thread_start_us":352,"threads_started":1}
I20260620 07:08:17.150313 30502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819edf66218a476fbd60c2b2d909f229" candidate_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" is_pre_election: true
I20260620 07:08:17.150493 30763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819edf66218a476fbd60c2b2d909f229" candidate_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d93c61f79954fab997872de5aa7de73" is_pre_election: true
I20260620 07:08:17.150952 30502 raft_consensus.cc:2454] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be60e28ff4d9471e946e0d5a03dc3d1a in term 0.
I20260620 07:08:17.150993 30763 raft_consensus.cc:2454] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be60e28ff4d9471e946e0d5a03dc3d1a in term 0.
I20260620 07:08:17.151721 30569 leader_election.cc:302] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d93c61f79954fab997872de5aa7de73, be60e28ff4d9471e946e0d5a03dc3d1a; no voters:
I20260620 07:08:17.152165 30831 raft_consensus.cc:2781] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:17.152297 30831 raft_consensus.cc:482] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:17.152390 30831 raft_consensus.cc:3037] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:17.154616 30831 raft_consensus.cc:504] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:17.155203 30831 leader_election.cc:288] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [CANDIDATE]: Term 1 election: Requested vote from peers c632caf2fbe04c1daf6b0673bfa27d56 (127.11.81.193:33787), 2d93c61f79954fab997872de5aa7de73 (127.11.81.195:33565)
I20260620 07:08:17.155810 30502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819edf66218a476fbd60c2b2d909f229" candidate_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c632caf2fbe04c1daf6b0673bfa27d56"
I20260620 07:08:17.155862 30763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819edf66218a476fbd60c2b2d909f229" candidate_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d93c61f79954fab997872de5aa7de73"
I20260620 07:08:17.156057 30763 raft_consensus.cc:3037] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:17.156054 30502 raft_consensus.cc:3037] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:17.159222 30763 raft_consensus.cc:2454] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be60e28ff4d9471e946e0d5a03dc3d1a in term 1.
I20260620 07:08:17.159221 30502 raft_consensus.cc:2454] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be60e28ff4d9471e946e0d5a03dc3d1a in term 1.
I20260620 07:08:17.159766 30566 leader_election.cc:302] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d93c61f79954fab997872de5aa7de73, be60e28ff4d9471e946e0d5a03dc3d1a, c632caf2fbe04c1daf6b0673bfa27d56; no voters:
I20260620 07:08:17.160151 30831 raft_consensus.cc:2781] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:17.160595 30831 raft_consensus.cc:686] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 1 LEADER]: Becoming Leader. State: Replica: be60e28ff4d9471e946e0d5a03dc3d1a, State: Running, Role: LEADER
I20260620 07:08:17.161125 30831 consensus_queue.cc:237] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } }
I20260620 07:08:17.165644 30360 catalog_manager.cc:5495] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to be60e28ff4d9471e946e0d5a03dc3d1a (127.11.81.194). New cstate: current_term: 1 leader_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be60e28ff4d9471e946e0d5a03dc3d1a" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 33969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:17.248910 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.243012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.11.81.194:33969)
0620 07:08:17.243206 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.243210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.243229 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:08:17.243351 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.243355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.243454 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.243582 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:17.244125 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.244888 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.246386 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.246912 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.247026 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.247653 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:17.248226 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:17.248374 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.248534 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.248576 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:08:17.257866 30502 raft_consensus.cc:1261] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 1 FOLLOWER]: Refusing update from remote peer be60e28ff4d9471e946e0d5a03dc3d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:17.257866 30763 raft_consensus.cc:1261] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 1 FOLLOWER]: Refusing update from remote peer be60e28ff4d9471e946e0d5a03dc3d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:17.259003 30831 consensus_queue.cc:1035] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d93c61f79954fab997872de5aa7de73" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:17.259356 30837 consensus_queue.cc:1035] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c632caf2fbe04c1daf6b0673bfa27d56" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:17.288740 30360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:17.288980 30360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:17.292593 30360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15e4c73392eb4d78bf6ad569bdd167b9: Sending DeleteTablet for 3 replicas of tablet 819edf66218a476fbd60c2b2d909f229
I20260620 07:08:17.293887 30482 tablet_service.cc:1514] Processing DeleteTablet for tablet 819edf66218a476fbd60c2b2d909f229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:17 UTC) from {username='slave'} at 127.0.0.1:45888
I20260620 07:08:17.293883 30613 tablet_service.cc:1514] Processing DeleteTablet for tablet 819edf66218a476fbd60c2b2d909f229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:17 UTC) from {username='slave'} at 127.0.0.1:58920
I20260620 07:08:17.294019 30743 tablet_service.cc:1514] Processing DeleteTablet for tablet 819edf66218a476fbd60c2b2d909f229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:17 UTC) from {username='slave'} at 127.0.0.1:51034
I20260620 07:08:17.295976 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30420
I20260620 07:08:17.304759 30852 tablet_replica.cc:330] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: stopping tablet replica
I20260620 07:08:17.305284 30852 raft_consensus.cc:2229] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:17.305801 30852 raft_consensus.cc:2258] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:17.308368 30852 ts_tablet_manager.cc:1905] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:17.315896 30852 ts_tablet_manager.cc:1918] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:17.316207 30852 log.cc:1196] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/819edf66218a476fbd60c2b2d909f229
I20260620 07:08:17.316841 30852 ts_tablet_manager.cc:1939] T 819edf66218a476fbd60c2b2d909f229 P 2d93c61f79954fab997872de5aa7de73: Deleting consensus metadata
I20260620 07:08:17.319041 30348 catalog_manager.cc:4841] TS 2d93c61f79954fab997872de5aa7de73 (127.11.81.195:33565): tablet 819edf66218a476fbd60c2b2d909f229 (table test-table [id=23e427217dd64e64baffa130f99f75ba]) successfully deleted
I20260620 07:08:17.325421 30851 tablet_replica.cc:330] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: stopping tablet replica
I20260620 07:08:17.325991 30850 tablet_replica.cc:330] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: stopping tablet replica
I20260620 07:08:17.325976 30851 raft_consensus.cc:2229] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:17.326462 30850 raft_consensus.cc:2229] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:17.326551 30851 raft_consensus.cc:2258] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:17.327219 30850 raft_consensus.cc:2258] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:17.329272 30850 ts_tablet_manager.cc:1905] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:17.329272 30851 ts_tablet_manager.cc:1905] T 819edf66218a476fbd60c2b2d909f229 P c632caf2fbe04c1daf6b0673bfa27d56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:17.329383 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.297254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.11.81.193:33787)
0620 07:08:17.297466 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.297472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.320473 (+ 23001us) server_negotiation.cc:407] Connection header received
0620 07:08:17.320680 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.320687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.320817 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.320929 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:17.321615 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.322994 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.325151 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.325853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.325985 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.327632 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:17.327644 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:17.327661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:17.328456 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:17.328521 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:17.328547 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:17.328635 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.328896 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.329022 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:08:17.330719 30542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:17.335569 30850 ts_tablet_manager.cc:1918] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:17.335846 30850 log.cc:1196] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/819edf66218a476fbd60c2b2d909f229
I20260620 07:08:17.336457 30850 ts_tablet_manager.cc:1939] T 819edf66218a476fbd60c2b2d909f229 P be60e28ff4d9471e946e0d5a03dc3d1a: Deleting consensus metadata
I20260620 07:08:17.338469 30348 catalog_manager.cc:4841] TS be60e28ff4d9471e946e0d5a03dc3d1a (127.11.81.194:33969): tablet 819edf66218a476fbd60c2b2d909f229 (table test-table [id=23e427217dd64e64baffa130f99f75ba]) successfully deleted
I20260620 07:08:17.448179 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30420
W20260620 07:08:17.462592 30345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:33787 (error 108)
I20260620 07:08:17.463456 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30551
I20260620 07:08:17.465101 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.464251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45902)
0620 07:08:17.464838 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.464943 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":357,"threads_started":1}
W20260620 07:08:17.465516 30345 catalog_manager.cc:4584] TS c632caf2fbe04c1daf6b0673bfa27d56 (127.11.81.193:33787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 819edf66218a476fbd60c2b2d909f229: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
I20260620 07:08:17.471881 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.464453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.11.81.194:33969)
0620 07:08:17.465000 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.465006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.465025 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:08:17.465087 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.465093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.465207 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.465341 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:17.465859 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.466966 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.468365 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.469044 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.469185 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.470155 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:17.470176 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:17.470199 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:17.471020 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:17.471080 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:17.471106 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:17.471224 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.471391 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.471518 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260620 07:08:17.472585 30673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:17.519457 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.518902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45910)
0620 07:08:17.519144 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.519256 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 07:08:17.520534 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.520131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45924)
0620 07:08:17.520294 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.520372 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 07:08:17.581038 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.580550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45930)
0620 07:08:17.580792 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.580884 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 07:08:17.582140 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.581717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45932)
0620 07:08:17.581893 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.581967 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:08:17.587271 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30551
I20260620 07:08:17.604436 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30681
I20260620 07:08:17.611490 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.605163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.11.81.195:33565)
0620 07:08:17.605329 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:17.605334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:17.605355 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:08:17.605519 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:17.605523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:17.605608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:17.605698 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:17.606092 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.606792 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.608266 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:17.608657 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:17.608751 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:17.609942 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:17.609952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:17.609967 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:17.610722 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:17.610774 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:17.610797 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:17.610897 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:17.611063 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:08:17.611171 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:08:17.612143 30803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:17.662998 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.662426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45942)
0620 07:08:17.662686 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.662802 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 07:08:17.664268 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:17.663764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:33787 (local address 127.0.0.1:45952)
0620 07:08:17.663995 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:17.664091 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:33787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 07:08:17.700433 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30681
I20260620 07:08:17.714295 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30330
I20260620 07:08:17.715617 30391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:17.823419 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30330
2026-06-20T07:08:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:17 dist-test-slave-wvz3 krb5kdc[30862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:17 dist-test-slave-wvz3 krb5kdc[30862](info): set up 2 sockets
Jun 20 07:08:17 dist-test-slave-wvz3 krb5kdc[30862](info): commencing operation
krb5kdc: starting...
W20260620 07:08:18.855367 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:19 dist-test-slave-wvz3 krb5kdc[30862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:19.118986 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40547
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:19.452385 30875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:19.452694 30875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:19.452813 30875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:19.460216 30875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:19.460368 30875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:19.460425 30875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:19.460502 30875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:19.460566 30875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:19.470777 30875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:19.472218 30875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:19.473575 30875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:19.487363 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:19.488278 30880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:19.491595 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:19.497233 30875 server_base.cc:1034] running on GCE node
I20260620 07:08:19.498157 30875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:19.499804 30875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:19.501130 30875 hybrid_clock.cc:648] HybridClock initialized: now 1781939299501076 us; error 45 us; skew 500 ppm
I20260620 07:08:19.501528 30875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:19 dist-test-slave-wvz3 krb5kdc[30862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:19.504870 30875 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:19.506546 30875 webserver.cc:460] Webserver started at http://127.11.81.254:40627/ using document root <none> and password file <none>
I20260620 07:08:19.507354 30875 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:19.507483 30875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:19.507812 30875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:19.510993 30875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "cea12bb1c0a34cdc8fccc7c47634611c"
format_stamp: "Formatted at 2026-06-20 07:08:19 on dist-test-slave-wvz3"
server_key: "400fbb15c6b6030563ef804cf21c317b"
server_key_iv: "f88e82c28f471d9a11cb07558ac5d165"
server_key_version: "encryptionkey@0"
I20260620 07:08:19.511857 30875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "cea12bb1c0a34cdc8fccc7c47634611c"
format_stamp: "Formatted at 2026-06-20 07:08:19 on dist-test-slave-wvz3"
server_key: "400fbb15c6b6030563ef804cf21c317b"
server_key_iv: "f88e82c28f471d9a11cb07558ac5d165"
server_key_version: "encryptionkey@0"
I20260620 07:08:19.517402 30875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:19.521111 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:19.522588 30875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:08:19.522756 30875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "cea12bb1c0a34cdc8fccc7c47634611c"
format_stamp: "Formatted at 2026-06-20 07:08:19 on dist-test-slave-wvz3"
server_key: "400fbb15c6b6030563ef804cf21c317b"
server_key_iv: "f88e82c28f471d9a11cb07558ac5d165"
server_key_version: "encryptionkey@0"
I20260620 07:08:19.522950 30875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:19.540643 30875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:19.545231 30875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:19.545503 30875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:19.563611 30875 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:40547
I20260620 07:08:19.563632 30941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:40547 every 8 connection(s)
I20260620 07:08:19.564980 30875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:19.567119 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30875
I20260620 07:08:19.567355 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:19.567909 11591 external_mini_cluster.cc:1428] Setting key 6a25913fec9c292f49c5aa66d8361b51
I20260620 07:08:19.569484 30942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:08:19 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:19.583688 30942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: Bootstrap starting.
I20260620 07:08:19.586987 30942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:19.588107 30942 log.cc:824] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:19.591035 30942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: No bootstrap required, opened a new log
I20260620 07:08:19.591641 30945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:19.569924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.11.81.254:40547)
0620 07:08:19.570727 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:19.570739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:19.570779 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:08:19.572413 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:19.572435 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:19.572843 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:19.573196 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:19.575110 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:19.576635 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:19.578428 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:19.579161 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:19.579348 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:19.584311 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:19.584320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:19.584338 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:19.584384 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:19.587402 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:19.588159 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:19.588166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:19.588173 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:19.588383 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:19.588778 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:19.588784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:19.588788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:19.589343 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:19.589604 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:19.590072 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:08:19.590511 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":372,"threads_started":1}
I20260620 07:08:19.599267 30942 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } }
I20260620 07:08:19.599792 30942 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:19.599898 30942 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea12bb1c0a34cdc8fccc7c47634611c, State: Initialized, Role: FOLLOWER
I20260620 07:08:19.600572 30942 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } }
I20260620 07:08:19.600863 30942 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:19.600998 30942 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:19.601173 30942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:19.603333 30942 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } }
I20260620 07:08:19.603861 30942 leader_election.cc:302] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea12bb1c0a34cdc8fccc7c47634611c; no voters:
I20260620 07:08:19.604422 30942 leader_election.cc:288] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:19.604532 30947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:19.605263 30947 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [term 1 LEADER]: Becoming Leader. State: Replica: cea12bb1c0a34cdc8fccc7c47634611c, State: Running, Role: LEADER
I20260620 07:08:19.605921 30947 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } }
I20260620 07:08:19.606696 30942 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:19.608908 30949 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea12bb1c0a34cdc8fccc7c47634611c. Latest consensus state: current_term: 1 leader_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } } }
I20260620 07:08:19.609331 30949 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:19.609232 30948 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea12bb1c0a34cdc8fccc7c47634611c" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40547 } } }
I20260620 07:08:19.609498 30948 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:19.610800 30954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:19.615586 30954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:19.624639 30954 catalog_manager.cc:1347] Generated new cluster ID: 673d88c0ce344e65a80550563460a1cf
I20260620 07:08:19.624758 30954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:19.643110 30954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:19.644531 30954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:19.657603 30954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: Generated new TSK 0
I20260620 07:08:19.658615 30954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.81.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
Entry for principal kudu/127.11.81.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab.
I20260620 07:08:19.718099 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:20.053270 30968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:20.053577 30968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:20.053704 30968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:20.061290 30968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:20.061412 30968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:20.061559 30968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:20.071736 30968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.193.keytab
--principal=kudu/127.11.81.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:20.073119 30968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:20.074466 30968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:20.087968 30974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:20.088027 30976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:20.088004 30973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:20.089226 30968 server_base.cc:1034] running on GCE node
I20260620 07:08:20.089968 30968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:20.091029 30968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:20.092278 30968 hybrid_clock.cc:648] HybridClock initialized: now 1781939300092203 us; error 72 us; skew 500 ppm
I20260620 07:08:20.092680 30968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:20 dist-test-slave-wvz3 krb5kdc[30862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:20.096024 30968 init.cc:377] Logged in from keytab as kudu/127.11.81.193@KRBTEST.COM (short username kudu)
I20260620 07:08:20.097656 30968 webserver.cc:460] Webserver started at http://127.11.81.193:45981/ using document root <none> and password file <none>
I20260620 07:08:20.098376 30968 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:20.098501 30968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:20.098821 30968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:20.102037 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "5c44bfce2c07441aa90943bfd839c269"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "d5b99291932d87b7f7ec03118117d32d"
server_key_iv: "4d9482c365330f82dcb9631119dd540b"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.102856 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "5c44bfce2c07441aa90943bfd839c269"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "d5b99291932d87b7f7ec03118117d32d"
server_key_iv: "4d9482c365330f82dcb9631119dd540b"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.108664 30968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:08:20.113054 30983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.114643 30968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:08:20.114807 30968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "5c44bfce2c07441aa90943bfd839c269"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "d5b99291932d87b7f7ec03118117d32d"
server_key_iv: "4d9482c365330f82dcb9631119dd540b"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.115015 30968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:20.129361 30968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:20.134007 30968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:20.134308 30968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:20.135386 30968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:20.136773 30968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:20.136853 30968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.136974 30968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:20.137030 30968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.165957 30968 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:37351
I20260620 07:08:20.166000 31095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:37351 every 8 connection(s)
I20260620 07:08:20.167228 30968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:20.172973 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 30968
I20260620 07:08:20.173130 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:20.173552 11591 external_mini_cluster.cc:1428] Setting key ff93b8bbb907ad9dddc6293bab3df907
Jun 20 07:08:20 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.193@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:20.189026 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:20.169883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:46605 (local address 127.11.81.254:40547)
0620 07:08:20.170310 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:20.170314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:20.171352 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 07:08:20.173300 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:20.173315 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:20.173400 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:20.173500 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:20.175457 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.176487 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.178152 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.178689 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.178824 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:20.182655 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:20.182663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:20.182667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:20.182709 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:20.185569 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:20.186624 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:20.186628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:20.186630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:20.186759 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:20.187254 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:20.187257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:20.187259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:20.187585 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:20.187699 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:20.188561 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 07:08:20.188749 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260620 07:08:20.190392 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:20.170449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40547 (local address 127.11.81.193:46605)
0620 07:08:20.171157 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:20.171204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:20.172979 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:20.173769 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:20.173777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:20.174429 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:20.175214 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:20.175242 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.176656 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.176660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:20.178003 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:20.178015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.178983 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.178988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:20.179205 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:20.179922 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:20.179947 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:20.182367 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:20.185793 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:20.185805 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:20.185835 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:20.186484 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:20.186871 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:20.186875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:20.186879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:20.187145 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:20.187709 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:20.187717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:20.188182 (+ 465us) client_negotiation.cc:769] Sending connection context
0620 07:08:20.188474 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 07:08:20.189044 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260620 07:08:20.192049 31096 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40547
I20260620 07:08:20.192415 31096 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:20.193318 31096 heartbeater.cc:511] Master 127.11.81.254:40547 requested a full tablet report, sending...
I20260620 07:08:20.195693 30907 ts_manager.cc:194] Registered new tserver with Master: 5c44bfce2c07441aa90943bfd839c269 (127.11.81.193:37351)
I20260620 07:08:20.197276 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.193@KRBTEST.COM'} at 127.11.81.193:46605
WARNING: no policy specified for kudu/127.11.81.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
Entry for principal kudu/127.11.81.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab.
I20260620 07:08:20.231634 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:20.557725 31102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:20.558038 31102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:20.558167 31102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:20.565608 31102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:20.565737 31102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:20.565898 31102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:20.576025 31102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.194.keytab
--principal=kudu/127.11.81.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:20.577332 31102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:20.578613 31102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:20.591887 31107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:20.591812 31110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:20.591830 31108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:20.592692 31102 server_base.cc:1034] running on GCE node
I20260620 07:08:20.593312 31102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:20.594264 31102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:20.595497 31102 hybrid_clock.cc:648] HybridClock initialized: now 1781939300595430 us; error 49 us; skew 500 ppm
I20260620 07:08:20.595849 31102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:20 dist-test-slave-wvz3 krb5kdc[30862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:20.602934 31102 init.cc:377] Logged in from keytab as kudu/127.11.81.194@KRBTEST.COM (short username kudu)
I20260620 07:08:20.604959 31102 webserver.cc:460] Webserver started at http://127.11.81.194:42185/ using document root <none> and password file <none>
I20260620 07:08:20.605641 31102 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:20.605746 31102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:20.606017 31102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:20.609174 31102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "5971912df619494eb923ca87f54ed1e0"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "2de8c4903b5b8b05932a778b36c82fb2"
server_key_iv: "3663e7315321a3686eba9dbc7a94c2a2"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.609938 31102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "5971912df619494eb923ca87f54ed1e0"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "2de8c4903b5b8b05932a778b36c82fb2"
server_key_iv: "3663e7315321a3686eba9dbc7a94c2a2"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.615738 31102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 07:08:20.619431 31117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.620970 31102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:20.621129 31102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "5971912df619494eb923ca87f54ed1e0"
format_stamp: "Formatted at 2026-06-20 07:08:20 on dist-test-slave-wvz3"
server_key: "2de8c4903b5b8b05932a778b36c82fb2"
server_key_iv: "3663e7315321a3686eba9dbc7a94c2a2"
server_key_version: "encryptionkey@0"
I20260620 07:08:20.621325 31102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:20.635402 31102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:20.639389 31102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:20.639665 31102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:20.640628 31102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:20.641988 31102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:20.642060 31102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.642176 31102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:20.642231 31102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:20.673072 31102 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36427
I20260620 07:08:20.673130 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36427 every 8 connection(s)
I20260620 07:08:20.674418 31102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:20.676163 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31102
I20260620 07:08:20.676364 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:20.676877 11591 external_mini_cluster.cc:1428] Setting key 07c2eeba1171a12fb9005da11ce20598
Jun 20 07:08:20 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.194@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:20.696234 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:20.677334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:38023 (local address 127.11.81.254:40547)
0620 07:08:20.677542 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:20.677548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:20.678876 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 07:08:20.680837 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:20.680843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:20.680936 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:20.681041 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:20.683040 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.683839 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.685483 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.685933 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.686050 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:20.690086 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:20.690092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:20.690095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:20.690134 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:20.692773 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:20.693779 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:20.693784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:20.693787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:20.693935 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:20.694413 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:20.694416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:20.694418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:20.694694 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:20.694820 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:20.695768 (+ 948us) server_negotiation.cc:299] Negotiation successful
0620 07:08:20.695969 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:08:20.697758 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:20.677906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40547 (local address 127.11.81.194:38023)
0620 07:08:20.678701 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:20.678748 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:20.680554 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:20.681337 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:20.681346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:20.682028 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:20.682854 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:20.682885 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.684004 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.684010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:20.685324 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:20.685336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:20.686171 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:20.686178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:20.686428 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:20.687177 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:20.687203 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:20.689756 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:20.692969 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:20.692979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:20.692999 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:20.693629 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:20.694043 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:20.694048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:20.694051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:20.694313 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:20.694816 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:20.694824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:20.695338 (+ 514us) client_negotiation.cc:769] Sending connection context
0620 07:08:20.695683 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 07:08:20.696266 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":321,"threads_started":1}
I20260620 07:08:20.699543 31230 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40547
I20260620 07:08:20.699927 31230 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:20.700906 31230 heartbeater.cc:511] Master 127.11.81.254:40547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.81.195@KRBTEST.COM; defaulting to no policy
I20260620 07:08:20.702730 30907 ts_manager.cc:194] Registered new tserver with Master: 5971912df619494eb923ca87f54ed1e0 (127.11.81.194:36427)
I20260620 07:08:20.703733 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.194@KRBTEST.COM'} at 127.11.81.194:38023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
Entry for principal kudu/127.11.81.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab.
I20260620 07:08:20.734557 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:21.062225 31235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:21.062551 31235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:21.062677 31235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:21.070384 31235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:21.070550 31235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:21.070712 31235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:21.080963 31235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.195.keytab
--principal=kudu/127.11.81.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:21.082273 31235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:21.083627 31235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:21.097332 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:21.097306 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:21.097275 31240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:21.100848 31235 server_base.cc:1034] running on GCE node
I20260620 07:08:21.101537 31235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:21.102632 31235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:21.103837 31235 hybrid_clock.cc:648] HybridClock initialized: now 1781939301103773 us; error 57 us; skew 500 ppm
I20260620 07:08:21.104180 31235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:21 dist-test-slave-wvz3 krb5kdc[30862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:21.107573 31235 init.cc:377] Logged in from keytab as kudu/127.11.81.195@KRBTEST.COM (short username kudu)
I20260620 07:08:21.109340 31235 webserver.cc:460] Webserver started at http://127.11.81.195:46295/ using document root <none> and password file <none>
I20260620 07:08:21.110091 31235 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:21.110213 31235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:21.110540 31235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:21.113732 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "a5295d44393a465eb8c8a1187fa4db1f"
format_stamp: "Formatted at 2026-06-20 07:08:21 on dist-test-slave-wvz3"
server_key: "9f162cef4a3be0bf8a7a67cb99c1f23d"
server_key_iv: "b797ded737f62b13fc244e63165a6c51"
server_key_version: "encryptionkey@0"
I20260620 07:08:21.114586 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "a5295d44393a465eb8c8a1187fa4db1f"
format_stamp: "Formatted at 2026-06-20 07:08:21 on dist-test-slave-wvz3"
server_key: "9f162cef4a3be0bf8a7a67cb99c1f23d"
server_key_iv: "b797ded737f62b13fc244e63165a6c51"
server_key_version: "encryptionkey@0"
I20260620 07:08:21.120445 31235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:08:21.124541 31250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:21.126176 31235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:08:21.126367 31235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "a5295d44393a465eb8c8a1187fa4db1f"
format_stamp: "Formatted at 2026-06-20 07:08:21 on dist-test-slave-wvz3"
server_key: "9f162cef4a3be0bf8a7a67cb99c1f23d"
server_key_iv: "b797ded737f62b13fc244e63165a6c51"
server_key_version: "encryptionkey@0"
I20260620 07:08:21.126603 31235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:21.146992 31235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:21.151713 31235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:21.152029 31235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:21.153082 31235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:21.154592 31235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:21.154673 31235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:21.154796 31235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:21.154848 31235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:21.186874 31235 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:35603
I20260620 07:08:21.186934 31362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:35603 every 8 connection(s)
I20260620 07:08:21.188313 31235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:21.189919 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31235
I20260620 07:08:21.190140 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:21.190673 11591 external_mini_cluster.cc:1428] Setting key b53c06c56011ca95a0504de1b3ebd817
I20260620 07:08:21.201130 31096 heartbeater.cc:503] Master 127.11.81.254:40547 was elected leader, sending a full tablet report...
Jun 20 07:08:21 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.195@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:21.212653 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.191204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:54921 (local address 127.11.81.254:40547)
0620 07:08:21.191458 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.191465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.192869 (+ 1404us) server_negotiation.cc:407] Connection header received
0620 07:08:21.195111 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.195116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.195205 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.195343 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:21.197114 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.198012 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.200481 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.200963 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.201085 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.205722 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:21.205727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:21.205730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:21.205769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:21.208959 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.209855 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.209860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.209862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.210034 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.210536 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.210540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.210543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.210861 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:21.211005 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.211993 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.212255 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 07:08:21.213629 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.191854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40547 (local address 127.11.81.195:54921)
0620 07:08:21.192701 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.192760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.194818 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.195635 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.195650 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.196260 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:21.196905 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.196926 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.198251 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.198256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.200252 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.200274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.201308 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.201318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.201611 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.202473 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:08:21.202514 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:21.205413 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:21.209146 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:21.209155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:21.209172 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:21.209676 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:21.210157 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:08:21.210162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:08:21.210165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:08:21.210404 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:08:21.211001 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:21.211010 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:21.211510 (+ 500us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.211802 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.212415 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":336,"threads_started":1}
I20260620 07:08:21.215165 31363 heartbeater.cc:347] Connected to a master server at 127.11.81.254:40547
I20260620 07:08:21.215588 31363 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:21.216398 31363 heartbeater.cc:511] Master 127.11.81.254:40547 requested a full tablet report, sending...
I20260620 07:08:21.218199 30907 ts_manager.cc:194] Registered new tserver with Master: a5295d44393a465eb8c8a1187fa4db1f (127.11.81.195:35603)
I20260620 07:08:21.219238 30907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.81.195@KRBTEST.COM'} at 127.11.81.195:54921
I20260620 07:08:21.219732 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:21.237300 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.223998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.11.81.254:40547)
0620 07:08:21.224211 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.224217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.224564 (+ 347us) server_negotiation.cc:407] Connection header received
0620 07:08:21.224802 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.224806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.224886 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.224989 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:21.226419 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.227268 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.228663 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.229156 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.229265 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.231785 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:21.231790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:21.231794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:21.231834 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:21.234481 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.235217 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.235222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.235224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.235411 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.235793 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.235799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.235802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.236156 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:21.236387 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.236739 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.236971 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:08:21.241294 30907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:21.248334 30907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:21.277141 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.262249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.11.81.193:37351)
0620 07:08:21.267934 (+ 5685us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.267945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.267985 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:08:21.268111 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.268119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.268441 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.269484 (+ 1043us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:21.269770 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.271422 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.274195 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.275772 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.276249 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.276533 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.276623 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.276827 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5488,"thread_start_us":5319,"threads_started":1}
I20260620 07:08:21.277204 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.261583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:37351 (local address 127.0.0.1:33734)
0620 07:08:21.262373 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.262413 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.262607 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.268717 (+ 6110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.268724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.268760 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:21.269227 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.269247 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.271556 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.271562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.274015 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.274030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.276027 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.276038 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.276294 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.276351 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.276502 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.276708 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":18,"thread_start_us":1852,"threads_started":1}
I20260620 07:08:21.278049 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.261777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36427 (local address 127.0.0.1:38686)
0620 07:08:21.268271 (+ 6494us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.268293 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.268441 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.269577 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.269582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.269606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:21.270007 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.270019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.271994 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.271999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.274229 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.274241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.276094 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.276100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.277316 (+ 1216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.277349 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.277410 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.277543 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6366,"mutex_wait_us":26,"thread_start_us":6232,"threads_started":1}
I20260620 07:08:21.278465 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.262367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.11.81.194:36427)
0620 07:08:21.268948 (+ 6581us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.268957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.268987 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:08:21.269093 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.269103 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.269362 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.270211 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:21.270427 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.271846 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.274390 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.275813 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.277733 (+ 1920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.277949 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.278019 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.278188 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6400,"thread_start_us":6298,"threads_started":1}
I20260620 07:08:21.278677 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.262899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.11.81.195:35603)
0620 07:08:21.263564 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.263572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.269549 (+ 5977us) server_negotiation.cc:407] Connection header received
0620 07:08:21.269874 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.269879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.270054 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.270175 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:21.270876 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.272320 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.275477 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.276798 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.276922 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.277158 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.278157 (+ 999us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.278356 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":334,"threads_started":1}
I20260620 07:08:21.278776 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.262227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:35603 (local address 127.0.0.1:34936)
0620 07:08:21.269422 (+ 7195us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.269440 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.269670 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.270422 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.270426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.270446 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:21.270706 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.270713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.272492 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.272497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.275270 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.275312 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.277216 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.277224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.277626 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.277661 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.277744 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.277873 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7068,"thread_start_us":6891,"threads_started":1}
I20260620 07:08:21.281718 31031 tablet_service.cc:1467] Processing CreateTablet for tablet a3af17ab18324110a756f40f21cb3b4b (DEFAULT_TABLE table=test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:21.282119 31298 tablet_service.cc:1467] Processing CreateTablet for tablet a3af17ab18324110a756f40f21cb3b4b (DEFAULT_TABLE table=test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:21.282115 31165 tablet_service.cc:1467] Processing CreateTablet for tablet a3af17ab18324110a756f40f21cb3b4b (DEFAULT_TABLE table=test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:21.283388 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af17ab18324110a756f40f21cb3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:21.283389 31165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af17ab18324110a756f40f21cb3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:21.283401 31031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3af17ab18324110a756f40f21cb3b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:21.295854 31381 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Bootstrap starting.
I20260620 07:08:21.298619 31382 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Bootstrap starting.
I20260620 07:08:21.299513 31383 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Bootstrap starting.
I20260620 07:08:21.300375 31381 tablet_bootstrap.cc:654] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:21.301833 31381 log.cc:824] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:21.303442 31383 tablet_bootstrap.cc:654] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:21.303541 31382 tablet_bootstrap.cc:654] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:21.304703 31382 log.cc:824] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:21.304701 31383 log.cc:824] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:21.305403 31381 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: No bootstrap required, opened a new log
I20260620 07:08:21.305758 31381 ts_tablet_manager.cc:1397] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:08:21.307600 31383 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: No bootstrap required, opened a new log
I20260620 07:08:21.307806 31382 tablet_bootstrap.cc:492] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: No bootstrap required, opened a new log
I20260620 07:08:21.307844 31383 ts_tablet_manager.cc:1397] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 07:08:21.308061 31382 ts_tablet_manager.cc:1397] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 07:08:21.315963 31383 raft_consensus.cc:348] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.316416 31383 raft_consensus.cc:374] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:21.316540 31383 raft_consensus.cc:729] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5971912df619494eb923ca87f54ed1e0, State: Initialized, Role: FOLLOWER
I20260620 07:08:21.316808 31381 raft_consensus.cc:348] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.317287 31381 raft_consensus.cc:374] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:21.317384 31381 raft_consensus.cc:729] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5295d44393a465eb8c8a1187fa4db1f, State: Initialized, Role: FOLLOWER
I20260620 07:08:21.317230 31383 consensus_queue.cc:260] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.318032 31381 consensus_queue.cc:260] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.318813 31382 raft_consensus.cc:348] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.319350 31382 raft_consensus.cc:374] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:21.319509 31382 raft_consensus.cc:729] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c44bfce2c07441aa90943bfd839c269, State: Initialized, Role: FOLLOWER
I20260620 07:08:21.320019 31381 ts_tablet_manager.cc:1428] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 07:08:21.320284 31382 consensus_queue.cc:260] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.321137 31230 heartbeater.cc:503] Master 127.11.81.254:40547 was elected leader, sending a full tablet report...
I20260620 07:08:21.321177 31363 heartbeater.cc:503] Master 127.11.81.254:40547 was elected leader, sending a full tablet report...
I20260620 07:08:21.321810 31383 ts_tablet_manager.cc:1428] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 07:08:21.323220 31382 ts_tablet_manager.cc:1428] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260620 07:08:21.346635 31389 raft_consensus.cc:482] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:21.346969 31389 raft_consensus.cc:504] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.348783 31389 leader_election.cc:288] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5971912df619494eb923ca87f54ed1e0 (127.11.81.194:36427), a5295d44393a465eb8c8a1187fa4db1f (127.11.81.195:35603)
I20260620 07:08:21.357874 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.349308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:51557 (local address 127.11.81.195:35603)
0620 07:08:21.349514 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.349521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.350151 (+ 630us) server_negotiation.cc:407] Connection header received
0620 07:08:21.350489 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.350493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.350559 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.350668 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:21.351267 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.352483 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.355410 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.356667 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.356871 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.356942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.357426 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.357571 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:08:21.357867 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.349322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:39951 (local address 127.11.81.194:36427)
0620 07:08:21.349527 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.349534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.350655 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 07:08:21.350739 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.350743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.350814 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.350887 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:21.351525 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.352542 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.355425 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.356649 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.356821 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.356901 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.357426 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.357571 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:08:21.358023 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.349789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36427 (local address 127.11.81.193:39951)
0620 07:08:21.350533 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.350545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.350642 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.350925 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.350929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.350955 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:21.351350 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.351359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.352701 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.352705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.355193 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.355212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.356918 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.356925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.357178 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.357258 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.357354 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.357513 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":206,"thread_start_us":291,"threads_started":1}
I20260620 07:08:21.358042 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.349249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:35603 (local address 127.11.81.193:51557)
0620 07:08:21.350035 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:21.350057 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:21.350283 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:21.350700 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:21.350704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:21.350728 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:21.351111 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.351127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.352691 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.352698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.355193 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:21.355212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.356937 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.356942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:21.357178 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.357248 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 07:08:21.357354 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 07:08:21.357539 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":82,"thread_start_us":435,"threads_started":1}
I20260620 07:08:21.359159 31318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af17ab18324110a756f40f21cb3b4b" candidate_uuid: "5c44bfce2c07441aa90943bfd839c269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5295d44393a465eb8c8a1187fa4db1f" is_pre_election: true
I20260620 07:08:21.359157 31185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af17ab18324110a756f40f21cb3b4b" candidate_uuid: "5c44bfce2c07441aa90943bfd839c269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5971912df619494eb923ca87f54ed1e0" is_pre_election: true
I20260620 07:08:21.359684 31318 raft_consensus.cc:2454] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c44bfce2c07441aa90943bfd839c269 in term 0.
I20260620 07:08:21.359684 31185 raft_consensus.cc:2454] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c44bfce2c07441aa90943bfd839c269 in term 0.
I20260620 07:08:21.360363 30986 leader_election.cc:302] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5971912df619494eb923ca87f54ed1e0, 5c44bfce2c07441aa90943bfd839c269; no voters:
I20260620 07:08:21.360752 31389 raft_consensus.cc:2781] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:21.360898 31389 raft_consensus.cc:482] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:21.361027 31389 raft_consensus.cc:3037] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:21.363237 31389 raft_consensus.cc:504] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.363930 31389 leader_election.cc:288] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [CANDIDATE]: Term 1 election: Requested vote from peers 5971912df619494eb923ca87f54ed1e0 (127.11.81.194:36427), a5295d44393a465eb8c8a1187fa4db1f (127.11.81.195:35603)
I20260620 07:08:21.364490 31318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af17ab18324110a756f40f21cb3b4b" candidate_uuid: "5c44bfce2c07441aa90943bfd839c269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5295d44393a465eb8c8a1187fa4db1f"
I20260620 07:08:21.364499 31185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3af17ab18324110a756f40f21cb3b4b" candidate_uuid: "5c44bfce2c07441aa90943bfd839c269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5971912df619494eb923ca87f54ed1e0"
I20260620 07:08:21.364692 31318 raft_consensus.cc:3037] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:21.364688 31185 raft_consensus.cc:3037] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:21.366919 31318 raft_consensus.cc:2454] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c44bfce2c07441aa90943bfd839c269 in term 1.
I20260620 07:08:21.367110 31185 raft_consensus.cc:2454] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c44bfce2c07441aa90943bfd839c269 in term 1.
I20260620 07:08:21.367446 30987 leader_election.cc:302] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c44bfce2c07441aa90943bfd839c269, a5295d44393a465eb8c8a1187fa4db1f; no voters:
I20260620 07:08:21.367790 31389 raft_consensus.cc:2781] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:21.368297 31389 raft_consensus.cc:686] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 1 LEADER]: Becoming Leader. State: Replica: 5c44bfce2c07441aa90943bfd839c269, State: Running, Role: LEADER
I20260620 07:08:21.369097 31389 consensus_queue.cc:237] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } }
I20260620 07:08:21.373651 30907 catalog_manager.cc:5495] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c44bfce2c07441aa90943bfd839c269 (127.11.81.193). New cstate: current_term: 1 leader_uuid: "5c44bfce2c07441aa90943bfd839c269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c44bfce2c07441aa90943bfd839c269" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:21.406708 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.401197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.11.81.193:37351)
0620 07:08:21.401378 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.401383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.401399 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:08:21.401489 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.401493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.401584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.401708 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:21.402038 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.402750 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.404199 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.404637 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.404737 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.405381 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:21.406056 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:21.406200 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.406385 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.406484 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:08:21.415382 31318 raft_consensus.cc:1261] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 1 FOLLOWER]: Refusing update from remote peer 5c44bfce2c07441aa90943bfd839c269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:21.415382 31185 raft_consensus.cc:1261] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 1 FOLLOWER]: Refusing update from remote peer 5c44bfce2c07441aa90943bfd839c269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:21.416426 31392 consensus_queue.cc:1035] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5971912df619494eb923ca87f54ed1e0" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:21.416797 31389 consensus_queue.cc:1035] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5295d44393a465eb8c8a1187fa4db1f" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:08:21.419606 31097 tablet.cc:2369] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:08:21.426314 31231 tablet.cc:2369] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:08:21.429163 31397 mvcc.cc:205] Tried to move back new op lower bound from 7298823378586943488 to 7298823378412011520. Current Snapshot: MvccSnapshot[applied={T|T < 7298823378586943488}]
I20260620 07:08:21.431818 31395 mvcc.cc:205] Tried to move back new op lower bound from 7298823378586943488 to 7298823378412011520. Current Snapshot: MvccSnapshot[applied={T|T < 7298823378586943488 or (T in {7298823378586943488})}]
I20260620 07:08:21.444792 30907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:21.445605 30907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:21.449254 30907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cea12bb1c0a34cdc8fccc7c47634611c: Sending DeleteTablet for 3 replicas of tablet a3af17ab18324110a756f40f21cb3b4b
I20260620 07:08:21.450372 31031 tablet_service.cc:1514] Processing DeleteTablet for tablet a3af17ab18324110a756f40f21cb3b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:21 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:33734
I20260620 07:08:21.450446 31165 tablet_service.cc:1514] Processing DeleteTablet for tablet a3af17ab18324110a756f40f21cb3b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:21 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:38686
I20260620 07:08:21.451262 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet a3af17ab18324110a756f40f21cb3b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:21 UTC) from {username='kudu', principal='kudu/127.11.81.254@KRBTEST.COM'} at 127.0.0.1:34936
I20260620 07:08:21.452602 31406 tablet_replica.cc:330] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: stopping tablet replica
I20260620 07:08:21.452603 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30968
I20260620 07:08:21.453096 31406 raft_consensus.cc:2229] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:21.453603 31406 raft_consensus.cc:2258] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:21.454350 31405 tablet_replica.cc:330] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: stopping tablet replica
I20260620 07:08:21.454736 31405 raft_consensus.cc:2229] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:21.455403 31405 raft_consensus.cc:2258] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:21.456404 31406 ts_tablet_manager.cc:1905] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:21.457224 31405 ts_tablet_manager.cc:1905] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:21.463213 31406 ts_tablet_manager.cc:1918] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:21.463474 31406 log.cc:1196] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/a3af17ab18324110a756f40f21cb3b4b
I20260620 07:08:21.463989 31406 ts_tablet_manager.cc:1939] T a3af17ab18324110a756f40f21cb3b4b P 5971912df619494eb923ca87f54ed1e0: Deleting consensus metadata
I20260620 07:08:21.464375 31405 ts_tablet_manager.cc:1918] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:21.464586 31405 log.cc:1196] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-0/wal/wals/a3af17ab18324110a756f40f21cb3b4b
I20260620 07:08:21.465134 31405 ts_tablet_manager.cc:1939] T a3af17ab18324110a756f40f21cb3b4b P 5c44bfce2c07441aa90943bfd839c269: Deleting consensus metadata
I20260620 07:08:21.466079 30893 catalog_manager.cc:4841] TS 5971912df619494eb923ca87f54ed1e0 (127.11.81.194:36427): tablet a3af17ab18324110a756f40f21cb3b4b (table test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]) successfully deleted
I20260620 07:08:21.466921 30892 catalog_manager.cc:4841] TS 5c44bfce2c07441aa90943bfd839c269 (127.11.81.193:37351): tablet a3af17ab18324110a756f40f21cb3b4b (table test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]) successfully deleted
I20260620 07:08:21.467516 31407 tablet_replica.cc:330] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: stopping tablet replica
I20260620 07:08:21.467903 31407 raft_consensus.cc:2229] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:21.468348 31407 raft_consensus.cc:2258] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:21.470373 31407 ts_tablet_manager.cc:1905] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:08:21 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.193@KRBTEST.COM
I20260620 07:08:21.477398 31407 ts_tablet_manager.cc:1918] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:21.477561 31407 log.cc:1196] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/a3af17ab18324110a756f40f21cb3b4b
I20260620 07:08:21.478070 31407 ts_tablet_manager.cc:1939] T a3af17ab18324110a756f40f21cb3b4b P a5295d44393a465eb8c8a1187fa4db1f: Deleting consensus metadata
I20260620 07:08:21.480273 30894 catalog_manager.cc:4841] TS a5295d44393a465eb8c8a1187fa4db1f (127.11.81.195:35603): tablet a3af17ab18324110a756f40f21cb3b4b (table test-table [id=d7c3d27916ae4eca863b6b3adf5ec5c9]) successfully deleted
I20260620 07:08:21.486971 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.455187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.11.81.193:37351)
0620 07:08:21.455405 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.455412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.466882 (+ 11470us) server_negotiation.cc:407] Connection header received
0620 07:08:21.467077 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.467083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.467177 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.467311 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:21.469263 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.470418 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.472375 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.473074 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.473185 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.478681 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:21.478698 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:21.478716 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:21.478767 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:21.482956 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.483976 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.483986 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.483990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.484200 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.484769 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.484775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.484779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.485307 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:21.485604 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.486299 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.486570 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:08:21.488229 31091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:21.588581 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30968
I20260620 07:08:21.604264 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31102
Jun 20 07:08:21 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.194@KRBTEST.COM
I20260620 07:08:21.619866 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.605029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.11.81.194:36427)
0620 07:08:21.605211 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.605217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.605392 (+ 175us) server_negotiation.cc:407] Connection header received
0620 07:08:21.605584 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.605590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.605677 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.605776 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:21.607610 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.608526 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.609761 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.610149 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.610258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.614437 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:21.614453 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:21.614469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:21.614508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:21.617151 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.617848 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.617855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.617858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.617983 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.618319 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.618329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.618331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.618709 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:21.618987 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.619274 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.619526 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:08:21.620460 31225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:21.708817 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31102
I20260620 07:08:21.723389 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31235
Jun 20 07:08:21 dist-test-slave-wvz3 krb5kdc[30862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.195@KRBTEST.COM
I20260620 07:08:21.738355 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:21.724139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.11.81.195:35603)
0620 07:08:21.724327 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:21.724331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:21.724350 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:08:21.724518 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:21.724522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:21.724596 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:21.724678 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:21.725947 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.726774 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.728196 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:21.728612 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:21.728711 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:21.733006 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:21.733022 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:21.733038 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:21.733076 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:21.735888 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.736597 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.736605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.736607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.736744 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:21.737064 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:21.737067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:21.737069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:21.737433 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:21.737686 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:21.737877 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:08:21.738048 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:08:21.739073 31358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:21.827425 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31235
I20260620 07:08:21.841078 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 30875
I20260620 07:08:21.842265 30937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:21.955375 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 30875
2026-06-20T07:08:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T07:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:22Z Disabled control of system clock
I20260620 07:08:22.028285 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39411
--webserver_interface=127.11.81.254
--webserver_port=0
--builtin_ntp_servers=127.11.81.212:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:22.358481 31416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:22.358794 31416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:22.358886 31416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:22.366276 31416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:22.366400 31416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:22.366436 31416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:22.366493 31416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:22.366535 31416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:22.375974 31416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:39411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:22.377243 31416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:22.378973 31416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:22.390527 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:22.390566 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:22.390542 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:22.391539 31416 server_base.cc:1034] running on GCE node
I20260620 07:08:22.392349 31416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:22.393898 31416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:22.395107 31416 hybrid_clock.cc:648] HybridClock initialized: now 1781939302395041 us; error 61 us; skew 500 ppm
I20260620 07:08:22.395484 31416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:22.397866 31416 webserver.cc:460] Webserver started at http://127.11.81.254:45259/ using document root <none> and password file <none>
I20260620 07:08:22.398567 31416 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:22.398667 31416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:22.398979 31416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:22.401986 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "5efc35c5361040ba895c8f87643d92e7"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "51170ae683f3829fe450e03fdff8190e"
server_key_iv: "5ca13377f167159818480060940b61ff"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.402798 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "5efc35c5361040ba895c8f87643d92e7"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "51170ae683f3829fe450e03fdff8190e"
server_key_iv: "5ca13377f167159818480060940b61ff"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.408093 31416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:08:22.411808 31430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:22.413218 31416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:08:22.413390 31416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "5efc35c5361040ba895c8f87643d92e7"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "51170ae683f3829fe450e03fdff8190e"
server_key_iv: "5ca13377f167159818480060940b61ff"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.413600 31416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:22.432336 31416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:22.433241 31416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:22.433511 31416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:22.451161 31416 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:39411
I20260620 07:08:22.451181 31481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:39411 every 8 connection(s)
I20260620 07:08:22.452540 31416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:22.453028 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31416
I20260620 07:08:22.453183 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:22.453773 11591 external_mini_cluster.cc:1428] Setting key 7b3d20cca9d9a8b5ce7aca15f5d23324
I20260620 07:08:22.456677 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:22.467619 31485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:22.455668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.11.81.254:39411)
0620 07:08:22.456627 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:22.456639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:22.456682 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:08:22.458342 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:22.458365 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:22.458856 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:22.459244 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:22.459905 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:22.461259 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:22.463093 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:22.463853 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:22.464003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:22.465171 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:22.465178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:22.465193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:22.465746 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:22.465825 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:22.465850 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:22.465956 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:22.466170 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:08:22.466512 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":345,"threads_started":1}
I20260620 07:08:22.469462 31482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: Bootstrap starting.
I20260620 07:08:22.472870 31482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:22.473948 31482 log.cc:824] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:22.477330 31482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: No bootstrap required, opened a new log
I20260620 07:08:22.485237 31482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } }
I20260620 07:08:22.485669 31482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:22.485754 31482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efc35c5361040ba895c8f87643d92e7, State: Initialized, Role: FOLLOWER
I20260620 07:08:22.486322 31482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } }
I20260620 07:08:22.486539 31482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:22.486645 31482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:22.486776 31482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:22.489099 31482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } }
I20260620 07:08:22.489699 31482 leader_election.cc:302] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efc35c5361040ba895c8f87643d92e7; no voters:
I20260620 07:08:22.490285 31482 leader_election.cc:288] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:22.490455 31487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:22.491125 31487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5efc35c5361040ba895c8f87643d92e7, State: Running, Role: LEADER
I20260620 07:08:22.491691 31487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } }
I20260620 07:08:22.492388 31482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:22.494848 31489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5efc35c5361040ba895c8f87643d92e7. Latest consensus state: current_term: 1 leader_uuid: "5efc35c5361040ba895c8f87643d92e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } } }
I20260620 07:08:22.495275 31489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:22.494731 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5efc35c5361040ba895c8f87643d92e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efc35c5361040ba895c8f87643d92e7" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 39411 } } }
I20260620 07:08:22.495990 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:22.498090 31495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:22.502753 31495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:22.512248 31495 catalog_manager.cc:1347] Generated new cluster ID: 450c64bc1d874764bbc3f307662ace71
I20260620 07:08:22.512391 31495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:22.523345 31495 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:22.525249 31495 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:22.543802 31495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: Generated new TSK 0
I20260620 07:08:22.545061 31495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:22.555738 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.193:0
--local_ip_for_outbound_sockets=127.11.81.193
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--builtin_ntp_servers=127.11.81.212:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:08:22.893877 31506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:22.894178 31506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:22.894268 31506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:22.901746 31506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:22.901868 31506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:22.901983 31506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.193
I20260620 07:08:22.911538 31506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.81.193
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.193
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:22.912878 31506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:22.914605 31506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:22.927639 31514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:22.927722 31511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:22.927695 31512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:22.928515 31506 server_base.cc:1034] running on GCE node
I20260620 07:08:22.929193 31506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:22.930189 31506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:22.931384 31506 hybrid_clock.cc:648] HybridClock initialized: now 1781939302931338 us; error 50 us; skew 500 ppm
I20260620 07:08:22.931736 31506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:22.934191 31506 webserver.cc:460] Webserver started at http://127.11.81.193:41995/ using document root <none> and password file <none>
I20260620 07:08:22.934918 31506 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:22.935024 31506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:22.935352 31506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:22.938817 31506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data/instance:
uuid: "d37786c0a4a24f5d95d914d53cadd329"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "0303d46d227947882e532145551d78d0"
server_key_iv: "af9e14e3a4e898525de88dc234004c2e"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.939713 31506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance:
uuid: "d37786c0a4a24f5d95d914d53cadd329"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "0303d46d227947882e532145551d78d0"
server_key_iv: "af9e14e3a4e898525de88dc234004c2e"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.945050 31506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:08:22.948822 31520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:22.950248 31506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:22.950419 31506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal
uuid: "d37786c0a4a24f5d95d914d53cadd329"
format_stamp: "Formatted at 2026-06-20 07:08:22 on dist-test-slave-wvz3"
server_key: "0303d46d227947882e532145551d78d0"
server_key_iv: "af9e14e3a4e898525de88dc234004c2e"
server_key_version: "encryptionkey@0"
I20260620 07:08:22.950628 31506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:22.972970 31506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:22.973853 31506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:22.974161 31506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:22.975174 31506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:22.976686 31506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:22.976780 31506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:22.976886 31506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:22.976958 31506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.007643 31506 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.193:35905
I20260620 07:08:23.007717 31632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.193:35905 every 8 connection(s)
I20260620 07:08:23.009027 31506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/data/info.pb
I20260620 07:08:23.010506 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31506
I20260620 07:08:23.010664 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-0/wal/instance
I20260620 07:08:23.011092 11591 external_mini_cluster.cc:1428] Setting key 2929fe4708536da204790b6f7f3752fa
I20260620 07:08:23.014461 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.194:0
--local_ip_for_outbound_sockets=127.11.81.194
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--builtin_ntp_servers=127.11.81.212:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:23.023859 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.013109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.193:35303 (local address 127.11.81.254:39411)
0620 07:08:23.014160 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.014164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.014875 (+ 711us) server_negotiation.cc:407] Connection header received
0620 07:08:23.016713 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.016719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.016798 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:23.016907 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:23.017991 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.018988 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.020806 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.021406 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.021528 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.022958 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:23.022963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:23.022967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:23.022987 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:23.023027 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:23.023039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:23.023118 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:23.023386 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 07:08:23.023496 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":862,"threads_started":1}
I20260620 07:08:23.024962 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.012518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39411 (local address 127.11.81.193:35303)
0620 07:08:23.014720 (+ 2202us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.014763 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.016452 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:23.017107 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:23.017116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:23.017212 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:23.017832 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.017850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.019217 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.019222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.020628 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.020642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.021634 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.021639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.021903 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.022598 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:23.022600 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:23.022644 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:23.022737 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:23.023132 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:23.023144 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:23.023153 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:08:23.023350 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 07:08:23.023791 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":551,"threads_started":1}
I20260620 07:08:23.026618 31633 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39411
I20260620 07:08:23.026930 31633 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:23.027731 31633 heartbeater.cc:511] Master 127.11.81.254:39411 requested a full tablet report, sending...
I20260620 07:08:23.029975 31447 ts_manager.cc:194] Registered new tserver with Master: d37786c0a4a24f5d95d914d53cadd329 (127.11.81.193:35905)
I20260620 07:08:23.032048 31447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.193:35303
W20260620 07:08:23.355057 31637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:23.355435 31637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:23.355543 31637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:23.363020 31637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:23.363135 31637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:23.363243 31637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.194
I20260620 07:08:23.372704 31637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.81.194
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.194
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:23.373975 31637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:23.375784 31637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:23.388391 31642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:23.388552 31643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:23.388439 31645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:23.389751 31637 server_base.cc:1034] running on GCE node
I20260620 07:08:23.390362 31637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:23.391378 31637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:23.392716 31637 hybrid_clock.cc:648] HybridClock initialized: now 1781939303392641 us; error 49 us; skew 500 ppm
I20260620 07:08:23.393044 31637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:23.395376 31637 webserver.cc:460] Webserver started at http://127.11.81.194:45445/ using document root <none> and password file <none>
I20260620 07:08:23.396075 31637 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:23.396178 31637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:23.396463 31637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:23.399462 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data/instance:
uuid: "bfc014122fbd4fcd8275ba5a1025ed1f"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "75e6cd514a6c1c9102e94b1918ee9e1b"
server_key_iv: "24c424ed55c2a8c4523e6900431c2065"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.400255 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance:
uuid: "bfc014122fbd4fcd8275ba5a1025ed1f"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "75e6cd514a6c1c9102e94b1918ee9e1b"
server_key_iv: "24c424ed55c2a8c4523e6900431c2065"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.405611 31637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:23.409369 31651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.410842 31637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:08:23.411053 31637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal
uuid: "bfc014122fbd4fcd8275ba5a1025ed1f"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "75e6cd514a6c1c9102e94b1918ee9e1b"
server_key_iv: "24c424ed55c2a8c4523e6900431c2065"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.411273 31637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:23.425675 31637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:23.426518 31637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:23.426805 31637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:23.427794 31637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:23.429253 31637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:23.429340 31637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.429452 31637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:23.429520 31637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.460357 31637 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.194:36901
I20260620 07:08:23.460389 31763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.194:36901 every 8 connection(s)
I20260620 07:08:23.461735 31637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/data/info.pb
I20260620 07:08:23.472038 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31637
I20260620 07:08:23.472205 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal/instance
I20260620 07:08:23.472743 11591 external_mini_cluster.cc:1428] Setting key 5fcce77b604636bb28c3613332c4b431
I20260620 07:08:23.475452 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.464385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:57265 (local address 127.11.81.254:39411)
0620 07:08:23.464591 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.464597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.466141 (+ 1544us) server_negotiation.cc:407] Connection header received
0620 07:08:23.468719 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.468723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.468797 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:23.468884 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:23.470023 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.470742 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.472434 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.473029 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.473171 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.474600 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:23.474604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:23.474607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:23.474625 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:23.474663 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:23.474679 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:23.474763 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:23.475013 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 07:08:23.475120 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:08:23.475796 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.81.195:0
--local_ip_for_outbound_sockets=127.11.81.195
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--builtin_ntp_servers=127.11.81.212:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:08:23.476549 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.464934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39411 (local address 127.11.81.194:57265)
0620 07:08:23.465963 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.466010 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.468403 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:23.469155 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:23.469164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:23.469265 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:23.469871 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.469891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.470865 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.470869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.472264 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.472282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.473239 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.473244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.473547 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.474266 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:23.474269 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:23.474311 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:23.474416 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:23.474777 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:23.474785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:23.474791 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:08:23.474951 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:08:23.475277 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260620 07:08:23.478174 31764 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39411
I20260620 07:08:23.478547 31764 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:23.479800 31764 heartbeater.cc:511] Master 127.11.81.254:39411 requested a full tablet report, sending...
I20260620 07:08:23.481657 31447 ts_manager.cc:194] Registered new tserver with Master: bfc014122fbd4fcd8275ba5a1025ed1f (127.11.81.194:36901)
I20260620 07:08:23.482728 31447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.194:57265
W20260620 07:08:23.817588 31767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:23.818005 31767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:23.818151 31767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:23.825533 31767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:23.825659 31767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:23.825817 31767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.81.195
I20260620 07:08:23.835668 31767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.81.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.81.195
--webserver_port=0
--tserver_master_addrs=127.11.81.254:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.81.195
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:23.837072 31767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:23.839006 31767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:23.852464 31772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:23.852294 31773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:23.852325 31775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:23.853201 31767 server_base.cc:1034] running on GCE node
I20260620 07:08:23.853870 31767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:23.854863 31767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:23.856081 31767 hybrid_clock.cc:648] HybridClock initialized: now 1781939303856016 us; error 51 us; skew 500 ppm
I20260620 07:08:23.856413 31767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:08:23.858781 31767 webserver.cc:460] Webserver started at http://127.11.81.195:32843/ using document root <none> and password file <none>
I20260620 07:08:23.859534 31767 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:23.859635 31767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:23.859964 31767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:23.862977 31767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data/instance:
uuid: "8333173fdf07449784c6de77a1de9630"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "e9c28e58b5fc29d1d85be56d57eedc44"
server_key_iv: "445ca511211a4d344413cb9a5f58dce6"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.863862 31767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance:
uuid: "8333173fdf07449784c6de77a1de9630"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "e9c28e58b5fc29d1d85be56d57eedc44"
server_key_iv: "445ca511211a4d344413cb9a5f58dce6"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.869205 31767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:08:23.873080 31781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.874703 31767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:08:23.874889 31767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal
uuid: "8333173fdf07449784c6de77a1de9630"
format_stamp: "Formatted at 2026-06-20 07:08:23 on dist-test-slave-wvz3"
server_key: "e9c28e58b5fc29d1d85be56d57eedc44"
server_key_iv: "445ca511211a4d344413cb9a5f58dce6"
server_key_version: "encryptionkey@0"
I20260620 07:08:23.875135 31767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:23.896147 31767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:23.896989 31767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:23.897269 31767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:23.898248 31767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:08:23.899747 31767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:08:23.899840 31767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.899946 31767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:08:23.900013 31767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:23.930372 31767 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.195:44777
I20260620 07:08:23.930429 31893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.195:44777 every 8 connection(s)
I20260620 07:08:23.931725 31767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/data/info.pb
I20260620 07:08:23.934557 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31767
I20260620 07:08:23.934723 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal/instance
I20260620 07:08:23.935326 11591 external_mini_cluster.cc:1428] Setting key c3e8a4729fd603fbf271cf477dc4f66e
I20260620 07:08:23.944743 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.934204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.195:41165 (local address 127.11.81.254:39411)
0620 07:08:23.934400 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.934405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.936145 (+ 1740us) server_negotiation.cc:407] Connection header received
0620 07:08:23.938419 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.938424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.938509 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:23.938591 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:23.939761 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.940495 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.942046 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.942461 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.942569 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.943942 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:23.943947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:23.943950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:23.943969 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:23.944001 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:23.944013 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:23.944085 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:23.944318 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 07:08:23.944421 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:08:23.945744 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.934871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:39411 (local address 127.11.81.195:41165)
0620 07:08:23.935977 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.936031 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.938135 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:23.938851 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:23.938859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:23.938952 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:08:23.939604 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.939623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.940640 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.940644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.941906 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.941918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.942680 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.942688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.942923 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.943616 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:08:23.943619 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:08:23.943659 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:08:23.943745 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:08:23.944099 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:08:23.944111 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:08:23.944118 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:08:23.944269 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 07:08:23.944684 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260620 07:08:23.947139 31894 heartbeater.cc:347] Connected to a master server at 127.11.81.254:39411
I20260620 07:08:23.947503 31894 heartbeater.cc:464] Registering TS with master...
I20260620 07:08:23.948269 31894 heartbeater.cc:511] Master 127.11.81.254:39411 requested a full tablet report, sending...
I20260620 07:08:23.949759 31447 ts_manager.cc:194] Registered new tserver with Master: 8333173fdf07449784c6de77a1de9630 (127.11.81.195:44777)
I20260620 07:08:23.950726 31447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.81.195:41165
I20260620 07:08:23.950779 11591 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:08:23.961509 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.955643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.11.81.254:39411)
0620 07:08:23.955795 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.955799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.956310 (+ 511us) server_negotiation.cc:407] Connection header received
0620 07:08:23.956415 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.956419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.956501 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:23.956591 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:23.957061 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.957775 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.959100 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.959617 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.959719 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.960772 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:23.960777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:23.960780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:23.960800 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:23.960836 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:23.960848 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:23.960927 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:23.961166 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 07:08:23.961266 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:08:23.965771 31447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:08:23.972729 31447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:08:23.997412 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.985561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.194:36901 (local address 127.0.0.1:52628)
0620 07:08:23.986599 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.986652 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.986834 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:23.988476 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:23.988486 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:23.988523 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:23.988954 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.988973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.991257 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.991263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.993978 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:23.994000 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.996242 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.996250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:23.996539 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.996609 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 07:08:23.996734 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 07:08:23.996973 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"threads_started":1}
I20260620 07:08:23.998973 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.986077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.11.81.194:36901)
0620 07:08:23.987380 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.987399 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.987433 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:08:23.987547 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.987556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.987826 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:23.989202 (+ 1376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:23.989445 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.991080 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.994289 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:23.995980 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:23.998043 (+ 2063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:23.998362 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:23.998449 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 07:08:23.998634 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1126,"thread_start_us":1013,"threads_started":1}
I20260620 07:08:24.003302 31699 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5c82e11799411e8e28fc542c6a538f (DEFAULT_TABLE table=test-table [id=a0a7171112f742768680ffbd58c57dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:24.004848 31699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5c82e11799411e8e28fc542c6a538f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:24.006091 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.985811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50192)
0620 07:08:23.997713 (+ 11902us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.997742 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.997865 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:23.999831 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:23.999837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:23.999880 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:24.000072 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.000083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.002171 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.002176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.003398 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.003416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.005176 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.005183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.005454 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.005490 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:08:24.005569 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:08:24.005707 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11763,"mutex_wait_us":21,"thread_start_us":319,"threads_started":1}
I20260620 07:08:24.007391 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.986014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.11.81.193:35905)
0620 07:08:23.999096 (+ 13082us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:23.999106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:23.999141 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:08:23.999258 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:23.999266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:23.999595 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.000276 (+ 681us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:24.000566 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.002008 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.003574 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.004996 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.006458 (+ 1462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.006664 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.006718 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.006907 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12902,"thread_start_us":13437,"threads_started":1}
I20260620 07:08:24.009745 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.987681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44777 (local address 127.0.0.1:43016)
0620 07:08:23.998572 (+ 10891us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:23.998590 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:23.999355 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:24.001237 (+ 1882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:24.001242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:24.001276 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:24.001649 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.001661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.003806 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.003811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.006255 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.006271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.008505 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.008511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.008693 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.008732 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:08:24.008798 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:08:24.008953 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"mutex_wait_us":33,"thread_start_us":13304,"threads_started":1}
I20260620 07:08:24.011602 31568 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5c82e11799411e8e28fc542c6a538f (DEFAULT_TABLE table=test-table [id=a0a7171112f742768680ffbd58c57dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:24.012450 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:23.988360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.11.81.195:44777)
0620 07:08:24.000299 (+ 11939us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.000308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.000344 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:08:24.000447 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.000455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.000700 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.001858 (+ 1158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:24.002132 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.003654 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.006479 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.008283 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.011367 (+ 3084us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.011698 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.011779 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.011991 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11750,"thread_start_us":11595,"threads_started":1}
I20260620 07:08:24.013267 31568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5c82e11799411e8e28fc542c6a538f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:24.016671 31829 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5c82e11799411e8e28fc542c6a538f (DEFAULT_TABLE table=test-table [id=a0a7171112f742768680ffbd58c57dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:08:24.018031 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5c82e11799411e8e28fc542c6a538f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:24.020370 31912 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Bootstrap starting.
I20260620 07:08:24.024937 31912 tablet_bootstrap.cc:654] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:24.026242 31912 log.cc:824] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:24.029564 31912 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: No bootstrap required, opened a new log
I20260620 07:08:24.029855 31912 ts_tablet_manager.cc:1397] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 07:08:24.029953 31914 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Bootstrap starting.
I20260620 07:08:24.031003 31915 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Bootstrap starting.
I20260620 07:08:24.033660 31914 tablet_bootstrap.cc:654] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:24.033974 31915 tablet_bootstrap.cc:654] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:24.034916 31915 log.cc:824] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:24.034978 31914 log.cc:824] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:24.035487 31633 heartbeater.cc:503] Master 127.11.81.254:39411 was elected leader, sending a full tablet report...
I20260620 07:08:24.041420 31915 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: No bootstrap required, opened a new log
I20260620 07:08:24.041054 31912 raft_consensus.cc:348] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.041574 31912 raft_consensus.cc:374] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:24.041682 31912 raft_consensus.cc:729] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc014122fbd4fcd8275ba5a1025ed1f, State: Initialized, Role: FOLLOWER
I20260620 07:08:24.041931 31915 ts_tablet_manager.cc:1397] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 07:08:24.042336 31912 consensus_queue.cc:260] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.044061 31764 heartbeater.cc:503] Master 127.11.81.254:39411 was elected leader, sending a full tablet report...
I20260620 07:08:24.044310 31914 tablet_bootstrap.cc:492] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: No bootstrap required, opened a new log
I20260620 07:08:24.044591 31914 ts_tablet_manager.cc:1397] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260620 07:08:24.044729 31912 ts_tablet_manager.cc:1428] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260620 07:08:24.051144 31915 raft_consensus.cc:348] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.051617 31915 raft_consensus.cc:374] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:24.051719 31915 raft_consensus.cc:729] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8333173fdf07449784c6de77a1de9630, State: Initialized, Role: FOLLOWER
I20260620 07:08:24.052345 31915 consensus_queue.cc:260] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.053541 31894 heartbeater.cc:503] Master 127.11.81.254:39411 was elected leader, sending a full tablet report...
I20260620 07:08:24.053442 31914 raft_consensus.cc:348] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.053912 31914 raft_consensus.cc:374] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:24.054019 31914 raft_consensus.cc:729] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37786c0a4a24f5d95d914d53cadd329, State: Initialized, Role: FOLLOWER
I20260620 07:08:24.054231 31915 ts_tablet_manager.cc:1428] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 07:08:24.054594 31914 consensus_queue.cc:260] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.056123 31914 ts_tablet_manager.cc:1428] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 07:08:24.088734 31918 raft_consensus.cc:482] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:08:24.089113 31918 raft_consensus.cc:504] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.090905 31918 leader_election.cc:288] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d37786c0a4a24f5d95d914d53cadd329 (127.11.81.193:35905), 8333173fdf07449784c6de77a1de9630 (127.11.81.195:44777)
I20260620 07:08:24.100127 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.091537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:41821 (local address 127.11.81.195:44777)
0620 07:08:24.091718 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.091734 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.092614 (+ 880us) server_negotiation.cc:407] Connection header received
0620 07:08:24.092745 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.092750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.092823 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.092903 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:24.093567 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.094608 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.097569 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.098859 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.099020 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.099092 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.099769 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.099892 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:08:24.100126 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.091536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.81.194:36603 (local address 127.11.81.193:35905)
0620 07:08:24.091753 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.091760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.092355 (+ 595us) server_negotiation.cc:407] Connection header received
0620 07:08:24.092554 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.092558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.092654 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.092763 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:08:24.093394 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.094597 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.097569 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.098859 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.099020 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.099092 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.099710 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.099855 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 07:08:24.100170 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.091173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.11.81.194:36603)
0620 07:08:24.092225 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.092243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:24.092414 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:24.092780 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:24.092783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:24.092803 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:24.093238 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.093250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.094743 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.094748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.097309 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.097322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.099263 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.099270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.099511 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.099565 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:08:24.099654 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:08:24.099820 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":226,"thread_start_us":1000,"threads_started":1}
I20260620 07:08:24.100734 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.091230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.195:44777 (local address 127.11.81.194:41821)
0620 07:08:24.092477 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.092511 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:08:24.092637 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:08:24.092943 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:08:24.092948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:08:24.092971 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:08:24.093425 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.093436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.094745 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.094749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.097299 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:08:24.097339 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.099250 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.099258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:08:24.099506 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.099557 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 07:08:24.099654 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:08:24.100504 (+ 850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":810,"thread_start_us":492,"threads_started":1}
I20260620 07:08:24.101181 31588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5c82e11799411e8e28fc542c6a538f" candidate_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37786c0a4a24f5d95d914d53cadd329" is_pre_election: true
I20260620 07:08:24.101825 31588 raft_consensus.cc:2454] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc014122fbd4fcd8275ba5a1025ed1f in term 0.
I20260620 07:08:24.101707 31849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5c82e11799411e8e28fc542c6a538f" candidate_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8333173fdf07449784c6de77a1de9630" is_pre_election: true
I20260620 07:08:24.102260 31849 raft_consensus.cc:2454] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc014122fbd4fcd8275ba5a1025ed1f in term 0.
I20260620 07:08:24.102560 31652 leader_election.cc:302] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc014122fbd4fcd8275ba5a1025ed1f, d37786c0a4a24f5d95d914d53cadd329; no voters:
I20260620 07:08:24.103008 31918 raft_consensus.cc:2781] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:08:24.103121 31918 raft_consensus.cc:482] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:08:24.103226 31918 raft_consensus.cc:3037] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:24.105444 31918 raft_consensus.cc:504] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.106027 31918 leader_election.cc:288] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [CANDIDATE]: Term 1 election: Requested vote from peers d37786c0a4a24f5d95d914d53cadd329 (127.11.81.193:35905), 8333173fdf07449784c6de77a1de9630 (127.11.81.195:44777)
I20260620 07:08:24.106568 31588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5c82e11799411e8e28fc542c6a538f" candidate_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37786c0a4a24f5d95d914d53cadd329"
I20260620 07:08:24.106570 31849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5c82e11799411e8e28fc542c6a538f" candidate_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8333173fdf07449784c6de77a1de9630"
I20260620 07:08:24.106757 31588 raft_consensus.cc:3037] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:24.106747 31849 raft_consensus.cc:3037] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:24.108981 31588 raft_consensus.cc:2454] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc014122fbd4fcd8275ba5a1025ed1f in term 1.
I20260620 07:08:24.108980 31849 raft_consensus.cc:2454] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc014122fbd4fcd8275ba5a1025ed1f in term 1.
I20260620 07:08:24.109450 31655 leader_election.cc:302] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8333173fdf07449784c6de77a1de9630, bfc014122fbd4fcd8275ba5a1025ed1f; no voters:
I20260620 07:08:24.109784 31918 raft_consensus.cc:2781] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:24.110194 31918 raft_consensus.cc:686] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 1 LEADER]: Becoming Leader. State: Replica: bfc014122fbd4fcd8275ba5a1025ed1f, State: Running, Role: LEADER
I20260620 07:08:24.110672 31918 consensus_queue.cc:237] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } }
I20260620 07:08:24.115773 31447 catalog_manager.cc:5495] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f reported cstate change: term changed from 0 to 1, leader changed from <none> to bfc014122fbd4fcd8275ba5a1025ed1f (127.11.81.194). New cstate: current_term: 1 leader_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc014122fbd4fcd8275ba5a1025ed1f" member_type: VOTER last_known_addr { host: "127.11.81.194" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 } health_report { overall_health: UNKNOWN } } }
I20260620 07:08:24.136539 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.131116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.11.81.194:36901)
0620 07:08:24.131380 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.131386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.131405 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:08:24.131611 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.131616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.131713 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.131882 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:08:24.132239 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.132946 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.134305 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.134725 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.134840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.135376 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:08:24.135939 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:08:24.136078 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.136162 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.136255 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:08:24.145206 31588 raft_consensus.cc:1261] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 1 FOLLOWER]: Refusing update from remote peer bfc014122fbd4fcd8275ba5a1025ed1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:24.145198 31849 raft_consensus.cc:1261] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 1 FOLLOWER]: Refusing update from remote peer bfc014122fbd4fcd8275ba5a1025ed1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:08:24.146202 31918 consensus_queue.cc:1035] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37786c0a4a24f5d95d914d53cadd329" member_type: VOTER last_known_addr { host: "127.11.81.193" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:24.146512 31923 consensus_queue.cc:1035] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8333173fdf07449784c6de77a1de9630" member_type: VOTER last_known_addr { host: "127.11.81.195" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:08:24.158743 31926 mvcc.cc:205] Tried to move back new op lower bound from 7298823389768306688 to 7298823389641904128. Current Snapshot: MvccSnapshot[applied={T|T < 7298823389768306688}]
I20260620 07:08:24.160142 31928 mvcc.cc:205] Tried to move back new op lower bound from 7298823389768306688 to 7298823389641904128. Current Snapshot: MvccSnapshot[applied={T|T < 7298823389768306688}]
I20260620 07:08:24.174453 31447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:24.174714 31447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:08:24.178282 31447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efc35c5361040ba895c8f87643d92e7: Sending DeleteTablet for 3 replicas of tablet 9e5c82e11799411e8e28fc542c6a538f
I20260620 07:08:24.179524 31699 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5c82e11799411e8e28fc542c6a538f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:24 UTC) from {username='slave'} at 127.0.0.1:52628
I20260620 07:08:24.179587 31568 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5c82e11799411e8e28fc542c6a538f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:24 UTC) from {username='slave'} at 127.0.0.1:50192
I20260620 07:08:24.179587 31829 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5c82e11799411e8e28fc542c6a538f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:08:24 UTC) from {username='slave'} at 127.0.0.1:43016
I20260620 07:08:24.180418 31936 tablet_replica.cc:330] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: stopping tablet replica
I20260620 07:08:24.180935 31936 raft_consensus.cc:2229] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:08:24.181704 31936 raft_consensus.cc:2258] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:24.183223 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31506
I20260620 07:08:24.184859 31936 ts_tablet_manager.cc:1905] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:24.193437 31936 ts_tablet_manager.cc:1918] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:24.193763 31936 log.cc:1196] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-1/wal/wals/9e5c82e11799411e8e28fc542c6a538f
I20260620 07:08:24.194442 31936 ts_tablet_manager.cc:1939] T 9e5c82e11799411e8e28fc542c6a538f P bfc014122fbd4fcd8275ba5a1025ed1f: Deleting consensus metadata
I20260620 07:08:24.196666 31433 catalog_manager.cc:4841] TS bfc014122fbd4fcd8275ba5a1025ed1f (127.11.81.194:36901): tablet 9e5c82e11799411e8e28fc542c6a538f (table test-table [id=a0a7171112f742768680ffbd58c57dfd]) successfully deleted
I20260620 07:08:24.200677 31937 tablet_replica.cc:330] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: stopping tablet replica
I20260620 07:08:24.201323 31937 raft_consensus.cc:2229] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:24.201973 31937 raft_consensus.cc:2258] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:24.204547 31937 ts_tablet_manager.cc:1905] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:24.205235 31938 tablet_replica.cc:330] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: stopping tablet replica
I20260620 07:08:24.205749 31938 raft_consensus.cc:2229] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:08:24.206276 31938 raft_consensus.cc:2258] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:08:24.208959 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.186590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.11.81.193:35905)
0620 07:08:24.186807 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.186815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.199923 (+ 13108us) server_negotiation.cc:407] Connection header received
0620 07:08:24.200196 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.200202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.200361 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.200489 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:24.201344 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.202679 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.204882 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.205472 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.205608 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.207186 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:24.207198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:24.207216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:24.208004 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:24.208065 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:24.208090 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:24.208185 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.208434 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.208559 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:08:24.208612 31938 ts_tablet_manager.cc:1905] T 9e5c82e11799411e8e28fc542c6a538f P d37786c0a4a24f5d95d914d53cadd329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:08:24.209754 31628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:24.213722 31937 ts_tablet_manager.cc:1918] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:08:24.214018 31937 log.cc:1196] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Deleting WAL directory at /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938992575046-11591-0/minicluster-data/ts-2/wal/wals/9e5c82e11799411e8e28fc542c6a538f
I20260620 07:08:24.214649 31937 ts_tablet_manager.cc:1939] T 9e5c82e11799411e8e28fc542c6a538f P 8333173fdf07449784c6de77a1de9630: Deleting consensus metadata
I20260620 07:08:24.217249 31434 catalog_manager.cc:4841] TS 8333173fdf07449784c6de77a1de9630 (127.11.81.195:44777): tablet 9e5c82e11799411e8e28fc542c6a538f (table test-table [id=a0a7171112f742768680ffbd58c57dfd]) successfully deleted
I20260620 07:08:24.340564 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31506
W20260620 07:08:24.362339 31431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.81.193:35905 (error 108)
I20260620 07:08:24.363083 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31637
I20260620 07:08:24.364497 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.363989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50202)
0620 07:08:24.364173 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.364305 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 07:08:24.365005 31431 catalog_manager.cc:4584] TS d37786c0a4a24f5d95d914d53cadd329 (127.11.81.193:35905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e5c82e11799411e8e28fc542c6a538f: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
I20260620 07:08:24.371634 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.363975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.11.81.194:36901)
0620 07:08:24.364251 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.364257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.364279 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:08:24.364449 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.364456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.364574 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.364700 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:24.365191 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.366406 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.367881 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.368572 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.368706 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.369823 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:24.369835 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:24.369855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:24.370666 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:24.370734 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:24.370770 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:24.370874 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.371043 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.371156 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260620 07:08:24.372442 31759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:24.418767 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.418240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50204)
0620 07:08:24.418477 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.418579 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 07:08:24.419993 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.419628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50214)
0620 07:08:24.419769 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.419848 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:08:24.480584 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.480087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50228)
0620 07:08:24.480301 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.480403 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:08:24.481597 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.481274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50242)
0620 07:08:24.481400 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.481472 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 07:08:24.486251 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31637
I20260620 07:08:24.503777 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31767
I20260620 07:08:24.511080 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.504599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.11.81.195:44777)
0620 07:08:24.504760 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:24.504764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:24.504782 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:08:24.505038 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:24.505042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:24.505137 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:24.505231 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:24.505714 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.506470 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.507923 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:24.508343 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:24.508452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:24.509609 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:24.509618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:24.509635 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:24.510291 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:24.510343 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:24.510365 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:24.510458 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:24.510656 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:08:24.510816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:08:24.511888 31889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:24.562484 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.561919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50256)
0620 07:08:24.562182 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.562285 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 07:08:24.563732 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:24.563276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.193:35905 (local address 127.0.0.1:50268)
0620 07:08:24.563526 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:08:24.563605 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.193:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 07:08:24.602150 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31767
I20260620 07:08:24.615917 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31416
I20260620 07:08:24.617215 31477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:24.734895 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31416
2026-06-20T07:08:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2783 ms)
[----------] 6 tests from EncryptionPolicyTest (21864 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:24 dist-test-slave-wvz3 krb5kdc[31947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:24 dist-test-slave-wvz3 krb5kdc[31947](info): set up 2 sockets
Jun 20 07:08:24 dist-test-slave-wvz3 krb5kdc[31947](info): commencing operation
krb5kdc: starting...
W20260620 07:08:25.787672 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:25 dist-test-slave-wvz3 krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:26.047622 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:26.386750 31960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:26.387072 31960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:26.387166 31960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:26.394630 31960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:26.394760 31960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:26.394807 31960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:26.394865 31960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:26.394917 31960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:26.405361 31960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:26.406726 31960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:26.408171 31960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:26.422192 31968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:26.422206 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:26.422200 31965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:26.423102 31960 server_base.cc:1034] running on GCE node
I20260620 07:08:26.423959 31960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:26.425592 31960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:26.426836 31960 hybrid_clock.cc:648] HybridClock initialized: now 1781939306426785 us; error 40 us; skew 500 ppm
I20260620 07:08:26.427220 31960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:26 dist-test-slave-wvz3 krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:26.438189 31960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:26.440070 31960 webserver.cc:460] Webserver started at http://127.0.0.1:36051/ using document root <none> and password file <none>
I20260620 07:08:26.440806 31960 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:26.440919 31960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:26.441198 31960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:26.444629 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "f1a3e2bc4c714c4e877bb96dc02c194a"
format_stamp: "Formatted at 2026-06-20 07:08:26 on dist-test-slave-wvz3"
server_key: "f829d6b0870311844646aa52451b6462"
server_key_iv: "67a889da8ea82abfd9d207764d6fbbd6"
server_key_version: "encryptionkey@0"
I20260620 07:08:26.445441 31960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "f1a3e2bc4c714c4e877bb96dc02c194a"
format_stamp: "Formatted at 2026-06-20 07:08:26 on dist-test-slave-wvz3"
server_key: "f829d6b0870311844646aa52451b6462"
server_key_iv: "67a889da8ea82abfd9d207764d6fbbd6"
server_key_version: "encryptionkey@0"
I20260620 07:08:26.450970 31960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:08:26.454780 31975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:26.456497 31960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:08:26.456696 31960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "f1a3e2bc4c714c4e877bb96dc02c194a"
format_stamp: "Formatted at 2026-06-20 07:08:26 on dist-test-slave-wvz3"
server_key: "f829d6b0870311844646aa52451b6462"
server_key_iv: "67a889da8ea82abfd9d207764d6fbbd6"
server_key_version: "encryptionkey@0"
I20260620 07:08:26.456924 31960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:26.482573 31960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:26.487119 31960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:26.487438 31960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:26.505522 31960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34091
I20260620 07:08:26.505545 32026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34091 every 8 connection(s)
I20260620 07:08:26.506937 31960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:26.511031 32027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:26.513720 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 31960
I20260620 07:08:26.513944 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:26.514583 11591 external_mini_cluster.cc:1428] Setting key d203fc9aad293bae6c6c80786f314e48
I20260620 07:08:26.524716 32027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a: Bootstrap starting.
I20260620 07:08:26.528683 32027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:26.529767 32027 log.cc:824] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:08:26 dist-test-slave-wvz3 krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:08:26.533041 32027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a: No bootstrap required, opened a new log
I20260620 07:08:26.540421 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:26.517004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33449 (local address 127.0.0.1:34091)
0620 07:08:26.517857 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:26.517874 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:26.517937 (+ 63us) server_negotiation.cc:407] Connection header received
0620 07:08:26.520210 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:26.520245 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:26.520889 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:26.521405 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:26.522821 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:26.524871 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:26.526281 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:26.527409 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:26.527598 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:26.532106 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:26.532115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:26.532132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:26.532171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:26.535768 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:26.536747 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:26.536755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:26.536762 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:26.536939 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:26.537448 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:26.537451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:26.537453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:26.538053 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:26.538391 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:26.538924 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 07:08:26.539261 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":372,"threads_started":1}
I20260620 07:08:26.541229 32027 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260620 07:08:26.541610 32027 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:26.541728 32027 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a3e2bc4c714c4e877bb96dc02c194a, State: Initialized, Role: FOLLOWER
I20260620 07:08:26.542418 32027 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260620 07:08:26.542722 32027 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:26.542860 32027 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:26.543023 32027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:26.544884 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:26.545835 32027 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260620 07:08:26.546339 32027 leader_election.cc:302] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a3e2bc4c714c4e877bb96dc02c194a; no voters:
I20260620 07:08:26.546965 32027 leader_election.cc:288] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:26.547125 32032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:26.547784 32032 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [term 1 LEADER]: Becoming Leader. State: Replica: f1a3e2bc4c714c4e877bb96dc02c194a, State: Running, Role: LEADER
I20260620 07:08:26.548554 32032 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } }
I20260620 07:08:26.551621 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260620 07:08:26.552062 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:26.553915 32032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a3e2bc4c714c4e877bb96dc02c194a. Latest consensus state: current_term: 1 leader_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a3e2bc4c714c4e877bb96dc02c194a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34091 } } }
I20260620 07:08:26.554143 32032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:26.559571 32027 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:26.559828 32037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:26.570983 32037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:26.579624 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:26.559891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36387 (local address 127.0.0.1:34091)
0620 07:08:26.560089 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:26.560095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:26.560668 (+ 573us) server_negotiation.cc:407] Connection header received
0620 07:08:26.560868 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:26.560874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:26.560973 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:26.561090 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:26.563488 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:26.564780 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:26.566799 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:26.567585 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:26.567775 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:26.571191 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:26.571198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:26.571203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:26.571246 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:26.575171 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:26.576361 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:26.576366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:26.576370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:26.576562 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:26.577152 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:26.577160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:26.577163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:26.577702 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:26.578859 (+ 1157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:26.579032 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:08:26.579209 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:08:26.585316 32037 catalog_manager.cc:1347] Generated new cluster ID: 1d0accaf5acc44b0bce1ccfba366eb92
I20260620 07:08:26.585547 32037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:26.597839 32037 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:26.599696 32037 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:26.607038 32037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a3e2bc4c714c4e877bb96dc02c194a: Generated new TSK 0
I20260620 07:08:26.608091 32037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:26.622853 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 31960
I20260620 07:08:26.624969 32022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:26.718381 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 31960
W20260620 07:08:26.732836 31945 connection.cc:535] client connection to 127.0.0.1:34091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34091): Connection reset by peer (error 104)
2026-06-20T07:08:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:26 dist-test-slave-wvz3 krb5kdc[32060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:26 dist-test-slave-wvz3 krb5kdc[32060](info): set up 2 sockets
Jun 20 07:08:26 dist-test-slave-wvz3 krb5kdc[32060](info): commencing operation
krb5kdc: starting...
W20260620 07:08:27.747535 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:27 dist-test-slave-wvz3 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:28.000605 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:28.340835 32073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:28.341153 32073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:28.341274 32073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:28.348806 32073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:28.348939 32073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:28.349004 32073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:28.349092 32073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:28.349152 32073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:28.359411 32073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:28.360803 32073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:28.362200 32073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:28.374253 32078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:28.374178 32079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:28.374209 32081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:28.375504 32073 server_base.cc:1034] running on GCE node
I20260620 07:08:28.376329 32073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:28.377851 32073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:28.379062 32073 hybrid_clock.cc:648] HybridClock initialized: now 1781939308379019 us; error 38 us; skew 500 ppm
I20260620 07:08:28.379462 32073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:28 dist-test-slave-wvz3 krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:28.382687 32073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:28.384305 32073 webserver.cc:460] Webserver started at http://127.0.0.1:38729/ using document root <none> and password file <none>
I20260620 07:08:28.385004 32073 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:28.385126 32073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:28.385416 32073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:28.388571 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "adfa5d431d3340febfa418815d39b183"
format_stamp: "Formatted at 2026-06-20 07:08:28 on dist-test-slave-wvz3"
server_key: "a55cfbf14e44455ad2d9c3986045273e"
server_key_iv: "029ec9cd43cb6d70df95b24f4dcb9eea"
server_key_version: "encryptionkey@0"
I20260620 07:08:28.389375 32073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "adfa5d431d3340febfa418815d39b183"
format_stamp: "Formatted at 2026-06-20 07:08:28 on dist-test-slave-wvz3"
server_key: "a55cfbf14e44455ad2d9c3986045273e"
server_key_iv: "029ec9cd43cb6d70df95b24f4dcb9eea"
server_key_version: "encryptionkey@0"
I20260620 07:08:28.394994 32073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:08:28.398838 32088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:28.400482 32073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:28.400679 32073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "adfa5d431d3340febfa418815d39b183"
format_stamp: "Formatted at 2026-06-20 07:08:28 on dist-test-slave-wvz3"
server_key: "a55cfbf14e44455ad2d9c3986045273e"
server_key_iv: "029ec9cd43cb6d70df95b24f4dcb9eea"
server_key_version: "encryptionkey@0"
I20260620 07:08:28.400895 32073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:28.416771 32073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:28.421316 32073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:28.421600 32073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:28.439764 32073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38325
I20260620 07:08:28.439792 32139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38325 every 8 connection(s)
I20260620 07:08:28.441224 32073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:28.445048 32140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:28.445968 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32073
I20260620 07:08:28.446152 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:28.446679 11591 external_mini_cluster.cc:1428] Setting key 8f76d1db646e6f70f8f3e9b24a6f0d14
I20260620 07:08:28.459551 32140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183: Bootstrap starting.
I20260620 07:08:28.461287 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:28.448599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38007 (local address 127.0.0.1:38325)
0620 07:08:28.449365 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:28.449378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:28.449424 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:08:28.451454 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:28.451500 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:28.451994 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:28.452387 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:28.453084 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:28.454443 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:28.456391 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:28.457192 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:28.457341 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:28.458739 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:28.458747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:28.458764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:28.459455 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:28.459548 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:28.459577 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:28.459736 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:28.460054 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 07:08:28.460261 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":349,"threads_started":1}
I20260620 07:08:28.462934 32140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:28.464105 32140 log.cc:824] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:28.465588 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:28.467254 32140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183: No bootstrap required, opened a new log
I20260620 07:08:28.477238 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:28.470079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58465 (local address 127.0.0.1:38325)
0620 07:08:28.470278 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:28.470284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:28.470524 (+ 240us) server_negotiation.cc:407] Connection header received
0620 07:08:28.470790 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:28.470795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:28.470866 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:28.470959 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:28.471626 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:28.472453 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:28.474390 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:28.474872 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:28.475032 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:28.476362 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:28.476367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:28.476370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:28.476390 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:28.476438 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:28.476458 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:28.476553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:28.476811 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:08:28.476886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 07:08:28.477216 32140 raft_consensus.cc:348] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260620 07:08:28.477778 32140 raft_consensus.cc:374] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:28.477903 32140 raft_consensus.cc:729] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfa5d431d3340febfa418815d39b183, State: Initialized, Role: FOLLOWER
I20260620 07:08:28.478658 32140 consensus_queue.cc:260] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260620 07:08:28.478965 32140 raft_consensus.cc:388] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:28.479118 32140 raft_consensus.cc:482] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:28.479317 32140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:28.481550 32140 raft_consensus.cc:504] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260620 07:08:28.482064 32140 leader_election.cc:302] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adfa5d431d3340febfa418815d39b183; no voters:
I20260620 07:08:28.482645 32140 leader_election.cc:288] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:28.482779 32150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:28.483489 32150 raft_consensus.cc:686] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [term 1 LEADER]: Becoming Leader. State: Replica: adfa5d431d3340febfa418815d39b183, State: Running, Role: LEADER
I20260620 07:08:28.484084 32150 consensus_queue.cc:237] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } }
I20260620 07:08:28.484933 32140 sys_catalog.cc:564] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:28.487429 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adfa5d431d3340febfa418815d39b183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } } }
I20260620 07:08:28.487411 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adfa5d431d3340febfa418815d39b183. Latest consensus state: current_term: 1 leader_uuid: "adfa5d431d3340febfa418815d39b183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adfa5d431d3340febfa418815d39b183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38325 } } }
I20260620 07:08:28.487869 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:28.487872 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:28.489544 32156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:28.494190 32156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:28.503930 32156 catalog_manager.cc:1347] Generated new cluster ID: 87c254e940904c1bb5ee87eec5a3af48
I20260620 07:08:28.504069 32156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:28.513382 32156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:28.515436 32156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:28.525156 32156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adfa5d431d3340febfa418815d39b183: Generated new TSK 0
I20260620 07:08:28.526225 32156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:28.543601 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32073
W20260620 07:08:28.545212 32135 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38007
W20260620 07:08:28.546116 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32073: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:28.546327 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32073
W20260620 07:08:28.560735 32058 connection.cc:535] client connection to 127.0.0.1:38325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38325): Connection reset by peer (error 104)
2026-06-20T07:08:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:28 dist-test-slave-wvz3 krb5kdc[32171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:28 dist-test-slave-wvz3 krb5kdc[32171](info): set up 2 sockets
Jun 20 07:08:28 dist-test-slave-wvz3 krb5kdc[32171](info): commencing operation
krb5kdc: starting...
W20260620 07:08:29.591500 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:29 dist-test-slave-wvz3 krb5kdc[32171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:29.843341 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:30.184653 32184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:30.185016 32184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:30.185160 32184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:30.192730 32184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:30.192884 32184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:30.192968 32184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:30.193055 32184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:30.193113 32184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:30.203989 32184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:30.205463 32184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:30.206875 32184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:30.219631 32189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:30.219534 32192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:30.219503 32190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:30.220870 32184 server_base.cc:1034] running on GCE node
I20260620 07:08:30.221887 32184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:30.223587 32184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:30.224892 32184 hybrid_clock.cc:648] HybridClock initialized: now 1781939310224783 us; error 42 us; skew 500 ppm
I20260620 07:08:30.225286 32184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:30 dist-test-slave-wvz3 krb5kdc[32171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:30.228724 32184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:30.230545 32184 webserver.cc:460] Webserver started at http://127.0.0.1:43789/ using document root <none> and password file <none>
I20260620 07:08:30.231272 32184 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:30.231457 32184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:30.231804 32184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:30.235018 32184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "f123a44e719e4c7fbdf91aa2697cf48b"
format_stamp: "Formatted at 2026-06-20 07:08:30 on dist-test-slave-wvz3"
server_key: "c2c842d3ebfa793278ac468381460827"
server_key_iv: "d70019712ea5aa8660bac6cc38bd7290"
server_key_version: "encryptionkey@0"
I20260620 07:08:30.235893 32184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "f123a44e719e4c7fbdf91aa2697cf48b"
format_stamp: "Formatted at 2026-06-20 07:08:30 on dist-test-slave-wvz3"
server_key: "c2c842d3ebfa793278ac468381460827"
server_key_iv: "d70019712ea5aa8660bac6cc38bd7290"
server_key_version: "encryptionkey@0"
I20260620 07:08:30.241554 32184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:30.245518 32199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:30.247076 32184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:08:30.247274 32184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "f123a44e719e4c7fbdf91aa2697cf48b"
format_stamp: "Formatted at 2026-06-20 07:08:30 on dist-test-slave-wvz3"
server_key: "c2c842d3ebfa793278ac468381460827"
server_key_iv: "d70019712ea5aa8660bac6cc38bd7290"
server_key_version: "encryptionkey@0"
I20260620 07:08:30.247525 32184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:30.253727 32184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:30.258432 32184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:30.258774 32184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:30.277586 32184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44377
I20260620 07:08:30.277621 32250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44377 every 8 connection(s)
I20260620 07:08:30.279055 32184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:30.282855 32251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:30.289088 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32184
I20260620 07:08:30.289276 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:30.289791 11591 external_mini_cluster.cc:1428] Setting key e8e268f9c1d0531852866ca9ab6c220d
I20260620 07:08:30.294359 32251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b: Bootstrap starting.
I20260620 07:08:30.297739 32251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:30.297849 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:30.291755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42775 (local address 127.0.0.1:44377)
0620 07:08:30.292610 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:30.292622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:30.292665 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:08:30.294368 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:30.294392 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:30.294788 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:30.295177 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:30.296246 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:30.296252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:30.296266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:30.296288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:30.296370 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:30.296393 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:30.296598 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:30.296736 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:08:30.296926 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":371,"threads_started":1}
I20260620 07:08:30.298859 32251 log.cc:824] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:30.301786 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:30.302067 32251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b: No bootstrap required, opened a new log
I20260620 07:08:30.308609 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:30.306221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54161 (local address 127.0.0.1:44377)
0620 07:08:30.306429 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:30.306435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:30.306637 (+ 202us) server_negotiation.cc:407] Connection header received
0620 07:08:30.306783 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:30.306789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:30.306861 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:30.306990 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:30.307843 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:30.307849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:30.307854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:30.307882 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:30.307937 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:30.307970 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:30.308114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:30.308230 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:08:30.308294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:08:30.310604 32251 raft_consensus.cc:348] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260620 07:08:30.310994 32251 raft_consensus.cc:374] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:30.311096 32251 raft_consensus.cc:729] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f123a44e719e4c7fbdf91aa2697cf48b, State: Initialized, Role: FOLLOWER
I20260620 07:08:30.311740 32251 consensus_queue.cc:260] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260620 07:08:30.311977 32251 raft_consensus.cc:388] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:30.312117 32251 raft_consensus.cc:482] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:30.312291 32251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:30.314709 32251 raft_consensus.cc:504] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260620 07:08:30.315235 32251 leader_election.cc:302] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f123a44e719e4c7fbdf91aa2697cf48b; no voters:
I20260620 07:08:30.315886 32251 leader_election.cc:288] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:30.316016 32261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:30.316635 32261 raft_consensus.cc:686] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [term 1 LEADER]: Becoming Leader. State: Replica: f123a44e719e4c7fbdf91aa2697cf48b, State: Running, Role: LEADER
I20260620 07:08:30.317191 32261 consensus_queue.cc:237] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } }
I20260620 07:08:30.317967 32251 sys_catalog.cc:564] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:30.320048 32262 sys_catalog.cc:455] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } } }
I20260620 07:08:30.320122 32263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f123a44e719e4c7fbdf91aa2697cf48b. Latest consensus state: current_term: 1 leader_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f123a44e719e4c7fbdf91aa2697cf48b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44377 } } }
I20260620 07:08:30.320420 32262 sys_catalog.cc:458] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:30.320447 32263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:30.322374 32267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:30.327562 32267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:30.337147 32267 catalog_manager.cc:1347] Generated new cluster ID: 3e5983d99e394334942087acc4d20f2b
I20260620 07:08:30.337284 32267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:30.357878 32267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:30.358398 32267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:30.364499 32267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f123a44e719e4c7fbdf91aa2697cf48b: Generated new TSK 0
I20260620 07:08:30.365675 32267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:30.370122 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32184
W20260620 07:08:30.371206 32246 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42775
W20260620 07:08:30.371976 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32184: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:30.372112 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32184
2026-06-20T07:08:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1825 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:30 dist-test-slave-wvz3 krb5kdc[32282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:30 dist-test-slave-wvz3 krb5kdc[32282](info): set up 2 sockets
Jun 20 07:08:30 dist-test-slave-wvz3 krb5kdc[32282](info): commencing operation
krb5kdc: starting...
W20260620 07:08:31.395545 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:31 dist-test-slave-wvz3 krb5kdc[32282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:31.659191 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:31.998606 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:31.998921 32295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:31.999068 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:32.006599 32295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:32.006726 32295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:32.006785 32295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:32.006845 32295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:32.006888 32295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:32.017135 32295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:32.018461 32295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:32.019775 32295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:32.031939 32300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:32.032015 32301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:32.031984 32303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:32.032961 32295 server_base.cc:1034] running on GCE node
I20260620 07:08:32.033797 32295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:32.035437 32295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:32.036681 32295 hybrid_clock.cc:648] HybridClock initialized: now 1781939312036627 us; error 42 us; skew 500 ppm
I20260620 07:08:32.037076 32295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:32 dist-test-slave-wvz3 krb5kdc[32282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:32.044216 32295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:32.045881 32295 webserver.cc:460] Webserver started at http://127.0.0.1:33893/ using document root <none> and password file <none>
I20260620 07:08:32.046626 32295 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:32.046748 32295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:32.047065 32295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:32.050333 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "85d3075a3ae24338b59de2ea2b76961d"
format_stamp: "Formatted at 2026-06-20 07:08:32 on dist-test-slave-wvz3"
server_key: "533df91dd1436416fd54f32d470f69f8"
server_key_iv: "4d4f28f30673be6f4a4389760b6ae429"
server_key_version: "encryptionkey@0"
I20260620 07:08:32.051172 32295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "85d3075a3ae24338b59de2ea2b76961d"
format_stamp: "Formatted at 2026-06-20 07:08:32 on dist-test-slave-wvz3"
server_key: "533df91dd1436416fd54f32d470f69f8"
server_key_iv: "4d4f28f30673be6f4a4389760b6ae429"
server_key_version: "encryptionkey@0"
I20260620 07:08:32.056941 32295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:08:32.060865 32310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:32.062566 32295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:32.062795 32295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "85d3075a3ae24338b59de2ea2b76961d"
format_stamp: "Formatted at 2026-06-20 07:08:32 on dist-test-slave-wvz3"
server_key: "533df91dd1436416fd54f32d470f69f8"
server_key_iv: "4d4f28f30673be6f4a4389760b6ae429"
server_key_version: "encryptionkey@0"
I20260620 07:08:32.063019 32295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:32.076150 32295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:32.080965 32295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:32.081264 32295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:32.099749 32295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39781
I20260620 07:08:32.099792 32361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39781 every 8 connection(s)
I20260620 07:08:32.101198 32295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:32.104846 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32295
I20260620 07:08:32.105034 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:32.105335 32362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:32.105567 11591 external_mini_cluster.cc:1428] Setting key 7917d337fb694e3cd77ed9076d2543d2
I20260620 07:08:32.119794 32362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d: Bootstrap starting.
Jun 20 07:08:32 dist-test-slave-wvz3 krb5kdc[32282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:08:32.123512 32362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:32.124626 32362 log.cc:824] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:32.128168 32362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d: No bootstrap required, opened a new log
I20260620 07:08:32.130393 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:32.107508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41427 (local address 127.0.0.1:39781)
0620 07:08:32.108310 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:32.108322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:32.108368 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:08:32.110280 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:32.110303 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:32.110734 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:32.111110 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:32.113038 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:32.114387 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:32.116215 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:32.117040 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:32.117201 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:32.122147 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:32.122160 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:32.122176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:32.122227 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:32.125687 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:32.126747 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:32.126755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:32.126762 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:32.126958 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:32.127505 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:32.127509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:32.127511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:32.128103 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:32.128381 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:32.128810 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 07:08:32.129260 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260620 07:08:32.134702 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:32.138160 32362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } }
I20260620 07:08:32.138744 32362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:32.138888 32362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d3075a3ae24338b59de2ea2b76961d, State: Initialized, Role: FOLLOWER
I20260620 07:08:32.139664 32362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } }
I20260620 07:08:32.139909 32362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:32.140045 32362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:32.140216 32362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:32.143478 32362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } }
I20260620 07:08:32.144062 32362 leader_election.cc:302] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d3075a3ae24338b59de2ea2b76961d; no voters:
I20260620 07:08:32.144763 32362 leader_election.cc:288] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:32.144865 32372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:32.146078 32372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [term 1 LEADER]: Becoming Leader. State: Replica: 85d3075a3ae24338b59de2ea2b76961d, State: Running, Role: LEADER
I20260620 07:08:32.146649 32372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } }
I20260620 07:08:32.147173 32362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:32.149881 32373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d3075a3ae24338b59de2ea2b76961d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } } }
I20260620 07:08:32.150306 32373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:32.152243 32374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d3075a3ae24338b59de2ea2b76961d. Latest consensus state: current_term: 1 leader_uuid: "85d3075a3ae24338b59de2ea2b76961d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d3075a3ae24338b59de2ea2b76961d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39781 } } }
I20260620 07:08:32.154629 32374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:32.158710 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:32.140242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41481 (local address 127.0.0.1:39781)
0620 07:08:32.140449 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:32.140456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:32.140937 (+ 481us) server_negotiation.cc:407] Connection header received
0620 07:08:32.141165 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:32.141169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:32.141245 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:32.141346 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:32.143224 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:32.144105 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:32.145915 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:32.146639 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:32.146752 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:32.150195 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:32.150202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:32.150207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:32.150250 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:32.154260 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:32.155825 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:32.155832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:32.155837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:32.156021 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:32.156554 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:32.156560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:32.156564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:32.157006 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:32.157271 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:32.157986 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 07:08:32.158278 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:08:32.161017 32378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:32.166379 32378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:32.167975 32389 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:08:32.176002 32378 catalog_manager.cc:1347] Generated new cluster ID: 53e8179a98e7434fa6b0aa66c6bb30b9
I20260620 07:08:32.176141 32378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:32.188905 32378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:32.190264 32378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:32.202636 32378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d3075a3ae24338b59de2ea2b76961d: Generated new TSK 0
I20260620 07:08:32.203629 32378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:32.209905 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32295
I20260620 07:08:32.211149 32357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:32.301688 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32295
2026-06-20T07:08:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:32 dist-test-slave-wvz3 krb5kdc[32394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:32 dist-test-slave-wvz3 krb5kdc[32394](info): set up 2 sockets
Jun 20 07:08:32 dist-test-slave-wvz3 krb5kdc[32394](info): commencing operation
krb5kdc: starting...
W20260620 07:08:33.323599 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:33 dist-test-slave-wvz3 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:33.582681 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:33.932566 32407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:33.932879 32407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:33.933080 32407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:33.940769 32407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:33.940905 32407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:33.940966 32407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:33.941056 32407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:33.941110 32407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:33.951783 32407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:33.953140 32407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:33.954470 32407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:33.966915 32415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:33.966821 32413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:33.967468 32412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:33.967727 32407 server_base.cc:1034] running on GCE node
I20260620 07:08:33.968575 32407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:33.970144 32407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:33.971447 32407 hybrid_clock.cc:648] HybridClock initialized: now 1781939313971382 us; error 44 us; skew 500 ppm
I20260620 07:08:33.971930 32407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:33 dist-test-slave-wvz3 krb5kdc[32394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:33.975320 32407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:33.977064 32407 webserver.cc:460] Webserver started at http://127.0.0.1:36783/ using document root <none> and password file <none>
I20260620 07:08:33.977775 32407 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:33.977910 32407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:33.978212 32407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:33.981431 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "46078cde32974415a2bc8d789073793b"
format_stamp: "Formatted at 2026-06-20 07:08:33 on dist-test-slave-wvz3"
server_key: "bc6ab338a8f9a6998255b9a131e12628"
server_key_iv: "b58ae3be76f9cbb69a376f47273f158a"
server_key_version: "encryptionkey@0"
I20260620 07:08:33.982261 32407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "46078cde32974415a2bc8d789073793b"
format_stamp: "Formatted at 2026-06-20 07:08:33 on dist-test-slave-wvz3"
server_key: "bc6ab338a8f9a6998255b9a131e12628"
server_key_iv: "b58ae3be76f9cbb69a376f47273f158a"
server_key_version: "encryptionkey@0"
I20260620 07:08:33.988175 32407 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 07:08:33.992193 32422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:33.993717 32407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:33.993911 32407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "46078cde32974415a2bc8d789073793b"
format_stamp: "Formatted at 2026-06-20 07:08:33 on dist-test-slave-wvz3"
server_key: "bc6ab338a8f9a6998255b9a131e12628"
server_key_iv: "b58ae3be76f9cbb69a376f47273f158a"
server_key_version: "encryptionkey@0"
I20260620 07:08:33.994125 32407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:34.008384 32407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:34.013031 32407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:34.013317 32407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:34.031884 32407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42611
I20260620 07:08:34.031951 32473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42611 every 8 connection(s)
I20260620 07:08:34.033315 32407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:34.037504 32474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:34.038571 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32407
I20260620 07:08:34.038717 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:34.039146 11591 external_mini_cluster.cc:1428] Setting key 9640991282d38cb3a87f938b1bcb0c02
I20260620 07:08:34.051107 32474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b: Bootstrap starting.
I20260620 07:08:34.052695 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:34.041090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59557 (local address 127.0.0.1:42611)
0620 07:08:34.041868 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:34.041882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:34.041923 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:08:34.043679 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:34.043703 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:34.044153 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:34.044501 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:34.045097 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:34.046315 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:34.048163 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:34.048893 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:34.049074 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:34.050316 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:34.050323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:34.050338 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:34.050889 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:34.050982 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:34.051008 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:34.051097 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:34.051376 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 07:08:34.051664 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":337,"threads_started":1}
I20260620 07:08:34.054356 32474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:34.055539 32474 log.cc:824] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:34.056326 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:34.058369 32474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b: No bootstrap required, opened a new log
I20260620 07:08:34.067929 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:34.060753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32975 (local address 127.0.0.1:42611)
0620 07:08:34.060943 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:34.060949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:34.061442 (+ 493us) server_negotiation.cc:407] Connection header received
0620 07:08:34.061774 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:34.061779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:34.061866 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:34.062003 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:34.062653 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:34.063450 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:34.065345 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:34.065827 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:34.065921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:34.067105 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:34.067110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:34.067114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:34.067135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:34.067181 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:34.067196 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:34.067279 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:34.067473 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 07:08:34.067558 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:08:34.068630 32474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260620 07:08:34.069031 32474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:34.069136 32474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46078cde32974415a2bc8d789073793b, State: Initialized, Role: FOLLOWER
I20260620 07:08:34.069777 32474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260620 07:08:34.070005 32474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:34.070138 32474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:34.070284 32474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:34.072751 32474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260620 07:08:34.073261 32474 leader_election.cc:302] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46078cde32974415a2bc8d789073793b; no voters:
I20260620 07:08:34.073841 32474 leader_election.cc:288] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:34.073972 32484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:34.074704 32484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [term 1 LEADER]: Becoming Leader. State: Replica: 46078cde32974415a2bc8d789073793b, State: Running, Role: LEADER
I20260620 07:08:34.075227 32484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260620 07:08:34.076042 32474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:34.078285 32485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46078cde32974415a2bc8d789073793b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } } }
I20260620 07:08:34.078722 32485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:34.079201 32486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46078cde32974415a2bc8d789073793b. Latest consensus state: current_term: 1 leader_uuid: "46078cde32974415a2bc8d789073793b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46078cde32974415a2bc8d789073793b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } } }
I20260620 07:08:34.079773 32486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:34.082496 32492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:34.089809 32492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:34.099208 32492 catalog_manager.cc:1347] Generated new cluster ID: e3a0b6f449e845af9cbd7a29864130a5
I20260620 07:08:34.099387 32492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:34.112082 32492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:34.113773 32492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:34.129886 32492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46078cde32974415a2bc8d789073793b: Generated new TSK 0
I20260620 07:08:34.130822 32492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:34.141763 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32407
W20260620 07:08:34.142913 32469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59557
W20260620 07:08:34.143832 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32407: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:34.144016 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32407
2026-06-20T07:08:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:34 dist-test-slave-wvz3 krb5kdc[32505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:34 dist-test-slave-wvz3 krb5kdc[32505](info): set up 2 sockets
Jun 20 07:08:34 dist-test-slave-wvz3 krb5kdc[32505](info): commencing operation
krb5kdc: starting...
W20260620 07:08:35.179620 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:35 dist-test-slave-wvz3 krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:35.436955 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:35.792662 32518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:35.792984 32518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:35.793093 32518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:35.800629 32518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:35.800776 32518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:35.800833 32518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:35.800901 32518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:35.800952 32518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:35.811573 32518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:35.812948 32518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:35.814327 32518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:35.827102 32523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:35.827169 32524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:35.827255 32526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:35.828796 32518 server_base.cc:1034] running on GCE node
I20260620 07:08:35.829774 32518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:35.831425 32518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:35.832696 32518 hybrid_clock.cc:648] HybridClock initialized: now 1781939315832649 us; error 43 us; skew 500 ppm
I20260620 07:08:35.833101 32518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:35 dist-test-slave-wvz3 krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:35.836714 32518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:35.838449 32518 webserver.cc:460] Webserver started at http://127.0.0.1:32861/ using document root <none> and password file <none>
I20260620 07:08:35.839172 32518 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:35.839316 32518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:35.839655 32518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:35.843061 32518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "b4e5e4d6477746838faea4ba1291b4a8"
format_stamp: "Formatted at 2026-06-20 07:08:35 on dist-test-slave-wvz3"
server_key: "e2006f0e5328b2e02ac966300a3aceac"
server_key_iv: "205eb392871100e0b219fc2bf420fc98"
server_key_version: "encryptionkey@0"
I20260620 07:08:35.843971 32518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "b4e5e4d6477746838faea4ba1291b4a8"
format_stamp: "Formatted at 2026-06-20 07:08:35 on dist-test-slave-wvz3"
server_key: "e2006f0e5328b2e02ac966300a3aceac"
server_key_iv: "205eb392871100e0b219fc2bf420fc98"
server_key_version: "encryptionkey@0"
I20260620 07:08:35.850535 32518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 07:08:35.854894 32533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:35.856606 32518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:08:35.856810 32518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "b4e5e4d6477746838faea4ba1291b4a8"
format_stamp: "Formatted at 2026-06-20 07:08:35 on dist-test-slave-wvz3"
server_key: "e2006f0e5328b2e02ac966300a3aceac"
server_key_iv: "205eb392871100e0b219fc2bf420fc98"
server_key_version: "encryptionkey@0"
I20260620 07:08:35.857035 32518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:35.863587 32518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:35.868435 32518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:35.868758 32518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:35.887435 32518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42329
I20260620 07:08:35.887492 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42329 every 8 connection(s)
I20260620 07:08:35.888942 32518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:35.893247 32585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:35.893383 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32518
I20260620 07:08:35.893561 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:35.894040 11591 external_mini_cluster.cc:1428] Setting key c82a4524790298ca00e34c1a2010e486
I20260620 07:08:35.903357 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:35.895929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38637 (local address 127.0.0.1:42329)
0620 07:08:35.896872 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:35.896885 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:35.896930 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:08:35.899504 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:35.899537 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:35.900052 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:35.900462 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:35.901632 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:35.901639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:35.901653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:35.901670 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:35.901733 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:35.901759 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:35.901989 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:35.902092 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 07:08:35.902304 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":400,"threads_started":1}
I20260620 07:08:35.907385 32585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8: Bootstrap starting.
I20260620 07:08:35.907508 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:35.910548 32585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:35.911924 32585 log.cc:824] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:35.914260 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:35.911826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33565 (local address 127.0.0.1:42329)
0620 07:08:35.912008 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:35.912015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:35.912406 (+ 391us) server_negotiation.cc:407] Connection header received
0620 07:08:35.912586 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:35.912590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:35.912652 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:35.912743 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:35.913639 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:35.913645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:35.913650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:35.913668 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:35.913705 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:35.913725 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:35.913826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:35.913931 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 07:08:35.913999 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:08:35.915117 32585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8: No bootstrap required, opened a new log
I20260620 07:08:35.923198 32585 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260620 07:08:35.923704 32585 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:35.923825 32585 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e5e4d6477746838faea4ba1291b4a8, State: Initialized, Role: FOLLOWER
I20260620 07:08:35.924423 32585 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260620 07:08:35.924636 32585 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:35.924782 32585 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:35.924935 32585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:35.927508 32585 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260620 07:08:35.928041 32585 leader_election.cc:302] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e5e4d6477746838faea4ba1291b4a8; no voters:
I20260620 07:08:35.928644 32585 leader_election.cc:288] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:35.928784 32595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:35.929579 32595 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [term 1 LEADER]: Becoming Leader. State: Replica: b4e5e4d6477746838faea4ba1291b4a8, State: Running, Role: LEADER
I20260620 07:08:35.930224 32595 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260620 07:08:35.931114 32585 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:35.932883 32596 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e5e4d6477746838faea4ba1291b4a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260620 07:08:35.933395 32596 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:35.933864 32597 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e5e4d6477746838faea4ba1291b4a8. Latest consensus state: current_term: 1 leader_uuid: "b4e5e4d6477746838faea4ba1291b4a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5e4d6477746838faea4ba1291b4a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260620 07:08:35.934180 32597 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:35.935946 32600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:35.941632 32600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:35.952059 32600 catalog_manager.cc:1347] Generated new cluster ID: b25167c693d5403597be9e267dcd09e9
I20260620 07:08:35.952186 32600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:35.972304 32600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:35.972904 32600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:35.981542 32600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e5e4d6477746838faea4ba1291b4a8: Generated new TSK 0
I20260620 07:08:35.982822 32600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:35.994225 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32518
W20260620 07:08:35.995476 32580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38637
W20260620 07:08:35.996402 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:35.996578 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32518
2026-06-20T07:08:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:36 dist-test-slave-wvz3 krb5kdc[32616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:36 dist-test-slave-wvz3 krb5kdc[32616](info): set up 2 sockets
Jun 20 07:08:36 dist-test-slave-wvz3 krb5kdc[32616](info): commencing operation
krb5kdc: starting...
W20260620 07:08:37.035590 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:37 dist-test-slave-wvz3 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:37.291317 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:37.634043 32629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:37.634373 32629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:37.634477 32629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:37.642141 32629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:37.642304 32629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:37.642361 32629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:37.642431 32629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:37.642482 32629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:37.653174 32629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:37.654523 32629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:37.655922 32629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:37.668442 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:37.668553 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:37.668744 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:37.670413 32629 server_base.cc:1034] running on GCE node
I20260620 07:08:37.671398 32629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:37.672937 32629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:37.674175 32629 hybrid_clock.cc:648] HybridClock initialized: now 1781939317674124 us; error 49 us; skew 500 ppm
I20260620 07:08:37.674543 32629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:37 dist-test-slave-wvz3 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:37.683167 32629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:37.684844 32629 webserver.cc:460] Webserver started at http://127.0.0.1:39743/ using document root <none> and password file <none>
I20260620 07:08:37.685587 32629 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:37.685725 32629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:37.686048 32629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:37.689301 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "8363a22c06d3438ab34f87d2bd9b6df3"
format_stamp: "Formatted at 2026-06-20 07:08:37 on dist-test-slave-wvz3"
server_key: "1faf39a8f4970243a7fae120445405fd"
server_key_iv: "9806f75bb12aa7d96252d1a137064db6"
server_key_version: "encryptionkey@0"
I20260620 07:08:37.690164 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "8363a22c06d3438ab34f87d2bd9b6df3"
format_stamp: "Formatted at 2026-06-20 07:08:37 on dist-test-slave-wvz3"
server_key: "1faf39a8f4970243a7fae120445405fd"
server_key_iv: "9806f75bb12aa7d96252d1a137064db6"
server_key_version: "encryptionkey@0"
I20260620 07:08:37.695827 32629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 07:08:37.699676 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:37.701161 32629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:37.701337 32629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "8363a22c06d3438ab34f87d2bd9b6df3"
format_stamp: "Formatted at 2026-06-20 07:08:37 on dist-test-slave-wvz3"
server_key: "1faf39a8f4970243a7fae120445405fd"
server_key_iv: "9806f75bb12aa7d96252d1a137064db6"
server_key_version: "encryptionkey@0"
I20260620 07:08:37.701545 32629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:37.726025 32629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:37.730562 32629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:37.730839 32629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:37.748615 32629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46177
I20260620 07:08:37.748644 32695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46177 every 8 connection(s)
I20260620 07:08:37.749987 32629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:37.754009 32696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:37.756491 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32629
I20260620 07:08:37.756682 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:37.757118 11591 external_mini_cluster.cc:1428] Setting key 35851382debd28698dd0cb0a6e7e2fd7
I20260620 07:08:37.766953 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3: Bootstrap starting.
I20260620 07:08:37.770568 32696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:37.771685 32696 log.cc:824] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:08:37 dist-test-slave-wvz3 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:08:37.775409 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3: No bootstrap required, opened a new log
I20260620 07:08:37.782398 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:37.759043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:55871 (local address 127.0.0.1:46177)
0620 07:08:37.759879 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:37.759896 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:37.759949 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:08:37.762208 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:37.762240 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:37.762783 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:37.763268 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:37.764596 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:37.766526 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:37.767889 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:37.768959 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:37.769151 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:37.773429 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:37.773440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:37.773462 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:37.773508 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:37.777370 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:37.778298 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:37.778308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:37.778317 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:37.778529 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:37.778976 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:37.778981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:37.778985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:37.779754 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:37.780076 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:37.780505 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 07:08:37.780891 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":355,"threads_started":1}
I20260620 07:08:37.785339 32696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260620 07:08:37.785799 32696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:37.785919 32696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8363a22c06d3438ab34f87d2bd9b6df3, State: Initialized, Role: FOLLOWER
I20260620 07:08:37.786512 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:37.786548 32696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260620 07:08:37.786823 32696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:37.786949 32696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:37.787104 32696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:37.789333 32696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260620 07:08:37.789875 32696 leader_election.cc:302] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8363a22c06d3438ab34f87d2bd9b6df3; no voters:
I20260620 07:08:37.790436 32696 leader_election.cc:288] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:37.790585 32705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:37.792902 32696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:37.797803 32705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [term 1 LEADER]: Becoming Leader. State: Replica: 8363a22c06d3438ab34f87d2bd9b6df3, State: Running, Role: LEADER
I20260620 07:08:37.798382 32705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } }
I20260620 07:08:37.801302 32706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260620 07:08:37.801733 32706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:37.802654 32707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8363a22c06d3438ab34f87d2bd9b6df3. Latest consensus state: current_term: 1 leader_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8363a22c06d3438ab34f87d2bd9b6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46177 } } }
I20260620 07:08:37.802932 32707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:37.813319 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:37.791545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:48839 (local address 127.0.0.1:46177)
0620 07:08:37.791734 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:37.791741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:37.794695 (+ 2954us) server_negotiation.cc:407] Connection header received
0620 07:08:37.794894 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:37.794900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:37.795007 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:37.795119 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:37.797339 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:37.798621 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:37.800433 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:37.801118 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:37.801225 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:37.804759 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:37.804767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:37.804772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:37.804819 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:37.808710 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:37.809783 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:37.809789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:37.809793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:37.809957 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:37.810519 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:37.810525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:37.810528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:37.811021 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:37.811322 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:37.812707 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0620 07:08:37.812919 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:08:37.821022 32716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:37.826093 32716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:37.835386 32716 catalog_manager.cc:1347] Generated new cluster ID: 0554d9a466084b2b8e7b52f972e8500d
I20260620 07:08:37.835518 32716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:37.854027 32716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:37.856021 32716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:37.865056 32716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8363a22c06d3438ab34f87d2bd9b6df3: Generated new TSK 0
I20260620 07:08:37.866181 32716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:37.884812 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32629
I20260620 07:08:37.886592 32691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:37.981036 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32629
W20260620 07:08:37.995201 32614 connection.cc:535] client connection to 127.0.0.1:46177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46177): Connection reset by peer (error 104)
2026-06-20T07:08:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:38 dist-test-slave-wvz3 krb5kdc[32729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:38 dist-test-slave-wvz3 krb5kdc[32729](info): set up 2 sockets
Jun 20 07:08:38 dist-test-slave-wvz3 krb5kdc[32729](info): commencing operation
krb5kdc: starting...
W20260620 07:08:39.021042 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:39 dist-test-slave-wvz3 krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:39.283124 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:39.614992 32742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:39.615386 32742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:39.615512 32742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:39.622841 32742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:39.622973 32742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:39.623026 32742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:39.623086 32742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:39.623129 32742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:39.633261 32742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:39.634526 32742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:39.635820 32742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:39.647629 32747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:39.647590 32748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:39.647660 32750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:39.648600 32742 server_base.cc:1034] running on GCE node
I20260620 07:08:39.649452 32742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:39.650971 32742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:39.652236 32742 hybrid_clock.cc:648] HybridClock initialized: now 1781939319652164 us; error 42 us; skew 500 ppm
I20260620 07:08:39.652652 32742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:39 dist-test-slave-wvz3 krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:39.656181 32742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:39.657853 32742 webserver.cc:460] Webserver started at http://127.0.0.1:36841/ using document root <none> and password file <none>
I20260620 07:08:39.658533 32742 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:39.658635 32742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:39.658893 32742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:39.662117 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "87d88712a7bf4d3fa73021498af451f0"
format_stamp: "Formatted at 2026-06-20 07:08:39 on dist-test-slave-wvz3"
server_key: "f353a4f9822af08f7b6b9958ecf4bc41"
server_key_iv: "cf1f60ac03ccb6fc7190118a9be1c0a0"
server_key_version: "encryptionkey@0"
I20260620 07:08:39.662904 32742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "87d88712a7bf4d3fa73021498af451f0"
format_stamp: "Formatted at 2026-06-20 07:08:39 on dist-test-slave-wvz3"
server_key: "f353a4f9822af08f7b6b9958ecf4bc41"
server_key_iv: "cf1f60ac03ccb6fc7190118a9be1c0a0"
server_key_version: "encryptionkey@0"
I20260620 07:08:39.668412 32742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:08:39.672245 32757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:39.673674 32742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:39.673844 32742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "87d88712a7bf4d3fa73021498af451f0"
format_stamp: "Formatted at 2026-06-20 07:08:39 on dist-test-slave-wvz3"
server_key: "f353a4f9822af08f7b6b9958ecf4bc41"
server_key_iv: "cf1f60ac03ccb6fc7190118a9be1c0a0"
server_key_version: "encryptionkey@0"
I20260620 07:08:39.674036 32742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:39.696552 32742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:39.701009 32742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:39.701287 32742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:39.718883 32742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45345
I20260620 07:08:39.718901 340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45345 every 8 connection(s)
I20260620 07:08:39.720351 32742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:39.724160 341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:39.727955 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 32742
I20260620 07:08:39.728147 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:39.728681 11591 external_mini_cluster.cc:1428] Setting key d9798ed3a800daa55141b372c6de966b
I20260620 07:08:39.736879 341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0: Bootstrap starting.
I20260620 07:08:39.740213 341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:39.741375 341 log.cc:824] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:39.742394 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:39.730744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:50453 (local address 127.0.0.1:45345)
0620 07:08:39.731626 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:39.731638 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:39.731680 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:08:39.733370 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:39.733396 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:39.733881 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:39.734277 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:39.734888 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:39.736264 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:39.737621 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:39.738309 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:39.738495 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:39.739792 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:39.739801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:39.739818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:39.740403 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:39.740493 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:39.740522 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:39.740696 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:39.741011 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 07:08:39.741198 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":343,"threads_started":1}
I20260620 07:08:39.744625 341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0: No bootstrap required, opened a new log
I20260620 07:08:39.746435 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:39.752950 341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 07:08:39.753377 341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:39.753494 341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d88712a7bf4d3fa73021498af451f0, State: Initialized, Role: FOLLOWER
I20260620 07:08:39.754063 341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 07:08:39.754287 341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:39.754413 341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:39.754585 341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:39.756985 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:39.750747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:39723 (local address 127.0.0.1:45345)
0620 07:08:39.750944 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:39.750952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:39.751354 (+ 402us) server_negotiation.cc:407] Connection header received
0620 07:08:39.751568 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:39.751572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:39.751643 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:39.751726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:39.752247 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:39.753010 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:39.754519 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:39.754999 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:39.755134 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:39.756284 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:39.756289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:39.756292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:39.756312 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:39.756370 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:39.756387 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:39.756482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:39.756668 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 07:08:39.756720 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:08:39.757274 341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 07:08:39.757819 341 leader_election.cc:302] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d88712a7bf4d3fa73021498af451f0; no voters:
I20260620 07:08:39.758468 341 leader_election.cc:288] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:39.758535 351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:39.759159 351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [term 1 LEADER]: Becoming Leader. State: Replica: 87d88712a7bf4d3fa73021498af451f0, State: Running, Role: LEADER
I20260620 07:08:39.759765 351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 07:08:39.760661 341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:39.762267 352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d88712a7bf4d3fa73021498af451f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260620 07:08:39.762622 352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:39.767542 353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d88712a7bf4d3fa73021498af451f0. Latest consensus state: current_term: 1 leader_uuid: "87d88712a7bf4d3fa73021498af451f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d88712a7bf4d3fa73021498af451f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260620 07:08:39.768004 353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:39.770447 356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:39.776219 356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:39.787266 356 catalog_manager.cc:1347] Generated new cluster ID: f6c95fbbbe0c4538933b52dea1d51309
I20260620 07:08:39.787465 356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:39.826702 356 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:39.828109 356 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:39.835011 356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d88712a7bf4d3fa73021498af451f0: Generated new TSK 0
I20260620 07:08:39.836093 356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:39.844336 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32742
W20260620 07:08:39.845793 336 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:50453
W20260620 07:08:39.846587 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 32742: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:39.846724 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 32742
W20260620 07:08:39.860947 32727 connection.cc:535] client connection to 127.0.0.1:45345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45345): Connection reset by peer (error 104)
2026-06-20T07:08:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1865 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:39 dist-test-slave-wvz3 krb5kdc[372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:39 dist-test-slave-wvz3 krb5kdc[372](info): set up 2 sockets
Jun 20 07:08:39 dist-test-slave-wvz3 krb5kdc[372](info): commencing operation
krb5kdc: starting...
W20260620 07:08:40.875818 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:41 dist-test-slave-wvz3 krb5kdc[372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:41.136982 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:41.475216 385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:41.475575 385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:41.475699 385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:41.485248 385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:41.485386 385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:41.485450 385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:41.485541 385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:41.485603 385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:41.496286 385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:41.497658 385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:41.499013 385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:41.511479 390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:41.511472 391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:41.511466 393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:41.512517 385 server_base.cc:1034] running on GCE node
I20260620 07:08:41.513406 385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:41.514950 385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:41.516182 385 hybrid_clock.cc:648] HybridClock initialized: now 1781939321516090 us; error 83 us; skew 500 ppm
I20260620 07:08:41.516561 385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:41 dist-test-slave-wvz3 krb5kdc[372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:41.519879 385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:41.521473 385 webserver.cc:460] Webserver started at http://127.0.0.1:34025/ using document root <none> and password file <none>
I20260620 07:08:41.522197 385 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:41.522315 385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:41.522614 385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:41.525854 385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "d28674bd305d4a12a480833728b411c7"
format_stamp: "Formatted at 2026-06-20 07:08:41 on dist-test-slave-wvz3"
server_key: "c8ef39d03e5e163a092fca2e8e0b927e"
server_key_iv: "0f65e1a2f3a1ff648d2d365720901772"
server_key_version: "encryptionkey@0"
I20260620 07:08:41.526674 385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "d28674bd305d4a12a480833728b411c7"
format_stamp: "Formatted at 2026-06-20 07:08:41 on dist-test-slave-wvz3"
server_key: "c8ef39d03e5e163a092fca2e8e0b927e"
server_key_iv: "0f65e1a2f3a1ff648d2d365720901772"
server_key_version: "encryptionkey@0"
I20260620 07:08:41.532414 385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:08:41.536100 400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:41.537585 385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:41.537827 385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "d28674bd305d4a12a480833728b411c7"
format_stamp: "Formatted at 2026-06-20 07:08:41 on dist-test-slave-wvz3"
server_key: "c8ef39d03e5e163a092fca2e8e0b927e"
server_key_iv: "0f65e1a2f3a1ff648d2d365720901772"
server_key_version: "encryptionkey@0"
I20260620 07:08:41.538038 385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:41.544184 385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:41.548645 385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:41.548944 385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:41.567113 385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39181
I20260620 07:08:41.567150 451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39181 every 8 connection(s)
I20260620 07:08:41.568792 385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:41.572728 452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:41.573038 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 385
I20260620 07:08:41.573196 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:41.573623 11591 external_mini_cluster.cc:1428] Setting key e2c513fa14743c102305e004a421b854
I20260620 07:08:41.582226 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:41.575508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:38847 (local address 127.0.0.1:39181)
0620 07:08:41.576346 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:41.576358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:41.576400 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:08:41.578695 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:41.578726 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:41.579164 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:41.579592 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:41.580643 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:41.580649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:41.580663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:41.580676 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:41.580731 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:41.580754 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:41.580928 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:41.581026 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 07:08:41.581221 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":372,"threads_started":1}
I20260620 07:08:41.586584 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:41.586927 452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7: Bootstrap starting.
I20260620 07:08:41.590334 452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:41.591830 452 log.cc:824] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:41.593699 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:41.591244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:50883 (local address 127.0.0.1:39181)
0620 07:08:41.591488 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:41.591494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:41.591870 (+ 376us) server_negotiation.cc:407] Connection header received
0620 07:08:41.592046 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:41.592050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:41.592101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:41.592185 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:41.593075 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:41.593081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:41.593086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:41.593103 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:41.593144 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:41.593159 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:41.593258 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:41.593374 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:08:41.593437 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 07:08:41.594877 452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7: No bootstrap required, opened a new log
I20260620 07:08:41.602746 452 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260620 07:08:41.603178 452 raft_consensus.cc:374] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:41.603257 452 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28674bd305d4a12a480833728b411c7, State: Initialized, Role: FOLLOWER
I20260620 07:08:41.604008 452 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260620 07:08:41.604239 452 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:41.604375 452 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:41.604527 452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:41.606846 452 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260620 07:08:41.607542 452 leader_election.cc:302] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28674bd305d4a12a480833728b411c7; no voters:
I20260620 07:08:41.608161 452 leader_election.cc:288] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:41.608255 462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:41.609000 462 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [term 1 LEADER]: Becoming Leader. State: Replica: d28674bd305d4a12a480833728b411c7, State: Running, Role: LEADER
I20260620 07:08:41.609666 462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260620 07:08:41.610432 452 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:41.612587 463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d28674bd305d4a12a480833728b411c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } } }
I20260620 07:08:41.612938 464 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28674bd305d4a12a480833728b411c7. Latest consensus state: current_term: 1 leader_uuid: "d28674bd305d4a12a480833728b411c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28674bd305d4a12a480833728b411c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } } }
I20260620 07:08:41.613180 463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:41.613268 464 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:41.616187 470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:41.621748 470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:41.631529 470 catalog_manager.cc:1347] Generated new cluster ID: 06e959fb1c014ee1828c721bc6f42b8f
I20260620 07:08:41.631662 470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:41.641615 470 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:41.642182 470 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:41.650406 470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d28674bd305d4a12a480833728b411c7: Generated new TSK 0
I20260620 07:08:41.651496 470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:41.666215 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 385
W20260620 07:08:41.667714 447 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:38847
W20260620 07:08:41.668543 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 385: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:41.668718 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 385
2026-06-20T07:08:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:41 dist-test-slave-wvz3 krb5kdc[483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:41 dist-test-slave-wvz3 krb5kdc[483](info): set up 2 sockets
Jun 20 07:08:41 dist-test-slave-wvz3 krb5kdc[483](info): commencing operation
krb5kdc: starting...
W20260620 07:08:42.695822 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:42 dist-test-slave-wvz3 krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:42.954478 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:43.292682 496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:43.293000 496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:43.293123 496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:43.300491 496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:43.300616 496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:43.300689 496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:43.300765 496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:43.300827 496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:43.310966 496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:43.312321 496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:43.313606 496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:43.325340 501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:43.325349 504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:43.325402 502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:43.326476 496 server_base.cc:1034] running on GCE node
I20260620 07:08:43.327347 496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:43.328828 496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:43.330047 496 hybrid_clock.cc:648] HybridClock initialized: now 1781939323330005 us; error 33 us; skew 500 ppm
I20260620 07:08:43.330400 496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:43 dist-test-slave-wvz3 krb5kdc[483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:43.338563 496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:43.340322 496 webserver.cc:460] Webserver started at http://127.0.0.1:41941/ using document root <none> and password file <none>
I20260620 07:08:43.341034 496 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:43.341161 496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:43.341455 496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:43.344583 496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "58c12a83b10043c4834b79f71ffae6a9"
format_stamp: "Formatted at 2026-06-20 07:08:43 on dist-test-slave-wvz3"
server_key: "b6b9defb6590aac7399d6f970378f300"
server_key_iv: "99e27dfdcfbf8187136bc12d19ec8a34"
server_key_version: "encryptionkey@0"
I20260620 07:08:43.345376 496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "58c12a83b10043c4834b79f71ffae6a9"
format_stamp: "Formatted at 2026-06-20 07:08:43 on dist-test-slave-wvz3"
server_key: "b6b9defb6590aac7399d6f970378f300"
server_key_iv: "99e27dfdcfbf8187136bc12d19ec8a34"
server_key_version: "encryptionkey@0"
I20260620 07:08:43.350688 496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:08:43.354502 511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:43.356079 496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:08:43.356258 496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "58c12a83b10043c4834b79f71ffae6a9"
format_stamp: "Formatted at 2026-06-20 07:08:43 on dist-test-slave-wvz3"
server_key: "b6b9defb6590aac7399d6f970378f300"
server_key_iv: "99e27dfdcfbf8187136bc12d19ec8a34"
server_key_version: "encryptionkey@0"
I20260620 07:08:43.356459 496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:43.378620 496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:43.383221 496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:43.383529 496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:43.401158 496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37005
I20260620 07:08:43.401178 562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37005 every 8 connection(s)
I20260620 07:08:43.402683 496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:43.406656 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:43.410275 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 496
I20260620 07:08:43.410445 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:43.410902 11591 external_mini_cluster.cc:1428] Setting key 9c93f4d14fba80ed13b745bd2952d92a
I20260620 07:08:43.419530 563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9: Bootstrap starting.
I20260620 07:08:43.422801 563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:43.423977 563 log.cc:824] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:08:43 dist-test-slave-wvz3 krb5kdc[483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:08:43.427048 563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9: No bootstrap required, opened a new log
I20260620 07:08:43.433887 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:43.412866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:57691 (local address 127.0.0.1:37005)
0620 07:08:43.413640 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:43.413652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:43.413696 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:08:43.415402 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:43.415433 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:43.415900 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:43.416258 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:43.418067 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:43.419570 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:43.420995 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:43.421708 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:43.421878 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:43.426097 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:43.426106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:43.426122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:43.426172 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:43.429425 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:43.430283 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:43.430291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:43.430297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:43.430472 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:43.430943 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:43.430951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:43.430953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:43.431571 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:43.431887 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:43.432442 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 07:08:43.432767 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260620 07:08:43.435356 563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260620 07:08:43.435735 563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:43.435828 563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c12a83b10043c4834b79f71ffae6a9, State: Initialized, Role: FOLLOWER
I20260620 07:08:43.436403 563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260620 07:08:43.436655 563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:43.436780 563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:43.436946 563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:43.437860 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:43.439368 563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260620 07:08:43.439870 563 leader_election.cc:302] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c12a83b10043c4834b79f71ffae6a9; no voters:
I20260620 07:08:43.440552 563 leader_election.cc:288] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:43.440668 569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:43.442577 563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:43.450371 569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 58c12a83b10043c4834b79f71ffae6a9, State: Running, Role: LEADER
I20260620 07:08:43.451149 569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260620 07:08:43.454591 573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c12a83b10043c4834b79f71ffae6a9. Latest consensus state: current_term: 1 leader_uuid: "58c12a83b10043c4834b79f71ffae6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } } }
I20260620 07:08:43.455006 573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:43.455457 572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58c12a83b10043c4834b79f71ffae6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c12a83b10043c4834b79f71ffae6a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } } }
I20260620 07:08:43.455763 572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:43.461117 589 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:08:43.462427 587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:43.467875 587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:43.470721 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:43.452229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:58925 (local address 127.0.0.1:37005)
0620 07:08:43.452435 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:43.452443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:43.452941 (+ 498us) server_negotiation.cc:407] Connection header received
0620 07:08:43.453153 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:43.453159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:43.453256 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:43.453391 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:43.455357 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:43.456620 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:43.458742 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:43.459651 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:43.459791 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:43.463158 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:43.463165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:43.463170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:43.463211 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:43.467017 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:43.468159 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:43.468165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:43.468169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:43.468332 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:43.468955 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:43.468961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:43.468965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:43.469390 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:43.469649 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:43.470121 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 07:08:43.470320 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:08:43.477852 587 catalog_manager.cc:1347] Generated new cluster ID: e8845334bd1741d3a9a0b84ae025770f
I20260620 07:08:43.478034 587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:43.494071 587 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:43.495544 587 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:43.502985 587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58c12a83b10043c4834b79f71ffae6a9: Generated new TSK 0
I20260620 07:08:43.503953 587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:43.511782 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 496
I20260620 07:08:43.513528 558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:43.604235 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 496
W20260620 07:08:43.618497 481 connection.cc:535] client connection to 127.0.0.1:37005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37005): Connection reset by peer (error 104)
2026-06-20T07:08:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:43 dist-test-slave-wvz3 krb5kdc[595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:43 dist-test-slave-wvz3 krb5kdc[595](info): set up 2 sockets
Jun 20 07:08:43 dist-test-slave-wvz3 krb5kdc[595](info): commencing operation
krb5kdc: starting...
W20260620 07:08:44.639838 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:44 dist-test-slave-wvz3 krb5kdc[595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:44.898864 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:45.238237 608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:45.238543 608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:45.238643 608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:45.245939 608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:45.246064 608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:45.246111 608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:45.246171 608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:45.246214 608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:45.256296 608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:45.257656 608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:45.258957 608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:45.271212 613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:45.271188 616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:45.271145 614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:45.272243 608 server_base.cc:1034] running on GCE node
I20260620 07:08:45.273010 608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:45.274518 608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:45.275740 608 hybrid_clock.cc:648] HybridClock initialized: now 1781939325275677 us; error 55 us; skew 500 ppm
I20260620 07:08:45.276096 608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:45 dist-test-slave-wvz3 krb5kdc[595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:45.279394 608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:45.280987 608 webserver.cc:460] Webserver started at http://127.0.0.1:39203/ using document root <none> and password file <none>
I20260620 07:08:45.281704 608 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:45.281817 608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:45.282109 608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:45.285303 608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "411adfa2cb6740fb849bd741f12793a7"
format_stamp: "Formatted at 2026-06-20 07:08:45 on dist-test-slave-wvz3"
server_key: "cf9cf0e693772c1784894a734511a945"
server_key_iv: "485e9737297536f1d38756dc3cfc81b7"
server_key_version: "encryptionkey@0"
I20260620 07:08:45.286092 608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "411adfa2cb6740fb849bd741f12793a7"
format_stamp: "Formatted at 2026-06-20 07:08:45 on dist-test-slave-wvz3"
server_key: "cf9cf0e693772c1784894a734511a945"
server_key_iv: "485e9737297536f1d38756dc3cfc81b7"
server_key_version: "encryptionkey@0"
I20260620 07:08:45.291714 608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:08:45.295507 623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:45.297128 608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:45.297343 608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "411adfa2cb6740fb849bd741f12793a7"
format_stamp: "Formatted at 2026-06-20 07:08:45 on dist-test-slave-wvz3"
server_key: "cf9cf0e693772c1784894a734511a945"
server_key_iv: "485e9737297536f1d38756dc3cfc81b7"
server_key_version: "encryptionkey@0"
I20260620 07:08:45.297562 608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:45.318924 608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:45.323601 608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:45.323892 608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:45.342669 608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39523
I20260620 07:08:45.342713 674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39523 every 8 connection(s)
I20260620 07:08:45.344300 608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:45.344743 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 608
I20260620 07:08:45.344879 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:45.345391 11591 external_mini_cluster.cc:1428] Setting key e5b6daccb95d063daea360596f3b836f
I20260620 07:08:45.349386 675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:45.360767 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:45.347415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:37113 (local address 127.0.0.1:39523)
0620 07:08:45.348171 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:45.348184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:45.348228 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:08:45.349970 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:45.349994 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:45.350430 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:45.350796 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:45.351686 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:45.353078 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:45.355278 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:45.356428 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:45.356584 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:45.358129 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:45.358137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:45.358153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:45.358752 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:45.358853 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:45.358881 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:45.359015 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:45.359392 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 07:08:45.359601 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260620 07:08:45.363250 675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7: Bootstrap starting.
I20260620 07:08:45.364840 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:45.367278 675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:45.368898 675 log.cc:824] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:45.372709 675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7: No bootstrap required, opened a new log
I20260620 07:08:45.377661 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:45.370324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:48729 (local address 127.0.0.1:39523)
0620 07:08:45.370510 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:45.370518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:45.371009 (+ 491us) server_negotiation.cc:407] Connection header received
0620 07:08:45.371221 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:45.371225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:45.371354 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:45.371441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:45.372103 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:45.372906 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:45.374794 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:45.375333 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:45.375443 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:45.376814 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:45.376820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:45.376823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:45.376847 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:45.376891 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:45.376910 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:45.377008 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:45.377236 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 07:08:45.377293 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:08:45.382270 675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } }
I20260620 07:08:45.382709 675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:45.382812 675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411adfa2cb6740fb849bd741f12793a7, State: Initialized, Role: FOLLOWER
I20260620 07:08:45.383472 675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } }
I20260620 07:08:45.383718 675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:45.383863 675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:45.384034 675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:45.387074 675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } }
I20260620 07:08:45.387718 675 leader_election.cc:302] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 411adfa2cb6740fb849bd741f12793a7; no voters:
I20260620 07:08:45.388566 675 leader_election.cc:288] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:45.388708 685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:45.389376 685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [term 1 LEADER]: Becoming Leader. State: Replica: 411adfa2cb6740fb849bd741f12793a7, State: Running, Role: LEADER
I20260620 07:08:45.389919 685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } }
I20260620 07:08:45.390796 675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:45.393347 686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "411adfa2cb6740fb849bd741f12793a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } } }
I20260620 07:08:45.393817 686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:45.394438 687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411adfa2cb6740fb849bd741f12793a7. Latest consensus state: current_term: 1 leader_uuid: "411adfa2cb6740fb849bd741f12793a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411adfa2cb6740fb849bd741f12793a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39523 } } }
I20260620 07:08:45.394846 687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:45.395526 694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:45.400354 694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:45.410419 694 catalog_manager.cc:1347] Generated new cluster ID: 4a78028058cc4355b3cfe0d8321919c5
I20260620 07:08:45.410562 694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:45.430730 694 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:45.432744 694 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:45.442193 694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411adfa2cb6740fb849bd741f12793a7: Generated new TSK 0
I20260620 07:08:45.443362 694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:45.457922 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 608
W20260620 07:08:45.459800 670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:37113
W20260620 07:08:45.460652 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 608: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:45.460847 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 608
W20260620 07:08:45.476024 593 connection.cc:535] client connection to 127.0.0.1:39523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39523): Connection reset by peer (error 104)
2026-06-20T07:08:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:45 dist-test-slave-wvz3 krb5kdc[706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:45 dist-test-slave-wvz3 krb5kdc[706](info): set up 2 sockets
Jun 20 07:08:45 dist-test-slave-wvz3 krb5kdc[706](info): commencing operation
krb5kdc: starting...
W20260620 07:08:46.483619 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:46 dist-test-slave-wvz3 krb5kdc[706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:08:46.741071 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:47.078639 719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:47.078959 719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:47.079097 719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:47.086673 719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:47.086824 719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:47.086906 719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:47.086983 719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:47.087046 719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:47.097487 719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:47.098861 719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:47.100214 719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:47.111950 725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:47.112057 727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:47.111979 724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:47.113334 719 server_base.cc:1034] running on GCE node
I20260620 07:08:47.114176 719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:47.115720 719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:47.116955 719 hybrid_clock.cc:648] HybridClock initialized: now 1781939327116906 us; error 39 us; skew 500 ppm
I20260620 07:08:47.117360 719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:47 dist-test-slave-wvz3 krb5kdc[706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:47.120723 719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:08:47.122367 719 webserver.cc:460] Webserver started at http://127.0.0.1:45855/ using document root <none> and password file <none>
I20260620 07:08:47.123051 719 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:47.123157 719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:47.123507 719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:47.126623 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "aecec21a19ab4fcf983af2c9f33df5cd"
format_stamp: "Formatted at 2026-06-20 07:08:47 on dist-test-slave-wvz3"
server_key: "7b92dc5d09f2043a1fb26c3926cb8754"
server_key_iv: "cd63167ab456a87f50a7ae42e7591407"
server_key_version: "encryptionkey@0"
I20260620 07:08:47.127465 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "aecec21a19ab4fcf983af2c9f33df5cd"
format_stamp: "Formatted at 2026-06-20 07:08:47 on dist-test-slave-wvz3"
server_key: "7b92dc5d09f2043a1fb26c3926cb8754"
server_key_iv: "cd63167ab456a87f50a7ae42e7591407"
server_key_version: "encryptionkey@0"
I20260620 07:08:47.133174 719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:08:47.137074 734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:47.138564 719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:08:47.138759 719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "aecec21a19ab4fcf983af2c9f33df5cd"
format_stamp: "Formatted at 2026-06-20 07:08:47 on dist-test-slave-wvz3"
server_key: "7b92dc5d09f2043a1fb26c3926cb8754"
server_key_iv: "cd63167ab456a87f50a7ae42e7591407"
server_key_version: "encryptionkey@0"
I20260620 07:08:47.138968 719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:47.145000 719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:47.149457 719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:47.149756 719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:47.167476 719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42019
I20260620 07:08:47.167505 785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42019 every 8 connection(s)
I20260620 07:08:47.169011 719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:47.172845 786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:47.175855 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 719
I20260620 07:08:47.176038 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:47.176533 11591 external_mini_cluster.cc:1428] Setting key 51b8f67723d82e10359846130ce1ad7e
I20260620 07:08:47.185050 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:47.178645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:39963 (local address 127.0.0.1:42019)
0620 07:08:47.179531 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:47.179544 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:47.179588 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:08:47.181348 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:47.181374 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:47.181778 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:47.182168 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:47.183376 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:47.183383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:47.183402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:47.183418 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:47.183482 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:47.183505 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:47.183696 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:47.183828 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 07:08:47.184023 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":348,"threads_started":1}
I20260620 07:08:47.185592 786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd: Bootstrap starting.
I20260620 07:08:47.189023 786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:47.189177 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:47.190083 786 log.cc:824] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:47.193245 786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd: No bootstrap required, opened a new log
I20260620 07:08:47.196779 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:47.193979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.148:51035 (local address 127.0.0.1:42019)
0620 07:08:47.194185 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:47.194191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:47.194583 (+ 392us) server_negotiation.cc:407] Connection header received
0620 07:08:47.194743 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:47.194748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:47.194818 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:47.194941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:47.195916 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:47.195923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:47.195928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:47.195947 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:47.196006 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:47.196026 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:47.196178 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:47.196338 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:08:47.196408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:08:47.201583 786 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:08:47.201998 786 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:47.202100 786 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecec21a19ab4fcf983af2c9f33df5cd, State: Initialized, Role: FOLLOWER
I20260620 07:08:47.202695 786 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:08:47.202916 786 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:47.203051 786 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:47.203219 786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:47.205469 786 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:08:47.205992 786 leader_election.cc:302] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecec21a19ab4fcf983af2c9f33df5cd; no voters:
I20260620 07:08:47.206693 786 leader_election.cc:288] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:47.206897 796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:47.208062 796 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [term 1 LEADER]: Becoming Leader. State: Replica: aecec21a19ab4fcf983af2c9f33df5cd, State: Running, Role: LEADER
I20260620 07:08:47.208731 796 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:08:47.208981 786 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:47.212275 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } } }
I20260620 07:08:47.212778 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:47.213174 798 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecec21a19ab4fcf983af2c9f33df5cd. Latest consensus state: current_term: 1 leader_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecec21a19ab4fcf983af2c9f33df5cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } } }
I20260620 07:08:47.213541 798 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:47.220762 805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:47.227068 805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:47.236905 805 catalog_manager.cc:1347] Generated new cluster ID: a0f38b5cb3d74f458094b41020019997
I20260620 07:08:47.237047 805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:47.251109 805 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:47.251701 805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:47.264060 805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecec21a19ab4fcf983af2c9f33df5cd: Generated new TSK 0
I20260620 07:08:47.264992 805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:47.278157 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 719
W20260620 07:08:47.279326 781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.148:39963
W20260620 07:08:47.280068 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:47.280203 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 719
2026-06-20T07:08:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1818 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:47 dist-test-slave-wvz3 krb5kdc[817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:47 dist-test-slave-wvz3 krb5kdc[817](info): set up 2 sockets
Jun 20 07:08:47 dist-test-slave-wvz3 krb5kdc[817](info): commencing operation
krb5kdc: starting...
W20260620 07:08:48.311684 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:48 dist-test-slave-wvz3 krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:48.566205 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35907
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:48.902428 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:48.902747 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:48.902866 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:48.910301 830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:48.910421 830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:48.910493 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:48.910569 830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:48.910626 830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:48.920832 830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:48.922158 830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:48.923517 830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:48.935308 835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:48.935355 838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:48.935415 836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:48.936347 830 server_base.cc:1034] running on GCE node
I20260620 07:08:48.937191 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:48.938621 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:48.939860 830 hybrid_clock.cc:648] HybridClock initialized: now 1781939328939799 us; error 48 us; skew 500 ppm
I20260620 07:08:48.940253 830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:48 dist-test-slave-wvz3 krb5kdc[817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939328, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:48.948645 830 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:48.950357 830 webserver.cc:460] Webserver started at http://127.11.81.254:46717/ using document root <none> and password file <none>
I20260620 07:08:48.951083 830 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:48.951206 830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:48.951550 830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:48.954658 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "fc64646255d74aa98dcb134871780343"
format_stamp: "Formatted at 2026-06-20 07:08:48 on dist-test-slave-wvz3"
server_key: "fcbb1e0d14f215a899a0994172044093"
server_key_iv: "35c2675ee993133ae72dc5a92ae0f622"
server_key_version: "encryptionkey@0"
I20260620 07:08:48.955461 830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "fc64646255d74aa98dcb134871780343"
format_stamp: "Formatted at 2026-06-20 07:08:48 on dist-test-slave-wvz3"
server_key: "fcbb1e0d14f215a899a0994172044093"
server_key_iv: "35c2675ee993133ae72dc5a92ae0f622"
server_key_version: "encryptionkey@0"
I20260620 07:08:48.960808 830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:08:48.964414 845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:48.965873 830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:08:48.966048 830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "fc64646255d74aa98dcb134871780343"
format_stamp: "Formatted at 2026-06-20 07:08:48 on dist-test-slave-wvz3"
server_key: "fcbb1e0d14f215a899a0994172044093"
server_key_iv: "35c2675ee993133ae72dc5a92ae0f622"
server_key_version: "encryptionkey@0"
I20260620 07:08:48.966233 830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:48.980020 830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:48.984416 830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:48.984697 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:49.003054 830 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35907
I20260620 07:08:49.003082 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35907 every 8 connection(s)
I20260620 07:08:49.004519 830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:49.008399 897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:49.010910 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 830
I20260620 07:08:49.011098 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:49.011662 11591 external_mini_cluster.cc:1428] Setting key d69134273ed83f82b38ab36b582e6ab9
I20260620 07:08:49.020318 897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343: Bootstrap starting.
I20260620 07:08:49.023422 897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:49.024423 897 log.cc:824] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:08:49 dist-test-slave-wvz3 krb5kdc[817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:49.027624 897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343: No bootstrap required, opened a new log
I20260620 07:08:49.036993 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:49.013512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51531 (local address 127.11.81.254:35907)
0620 07:08:49.014324 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:49.014337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:49.014381 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:08:49.016126 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:49.016150 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:49.016565 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:49.016930 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:49.018432 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:49.019792 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:49.021038 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:49.021730 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:49.021906 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:49.026355 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:49.026365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:49.026381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:49.026422 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:49.030180 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:49.031601 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:49.031612 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:49.031626 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:49.031890 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:49.032628 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:49.032634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:49.032636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:49.033527 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:49.033903 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:49.034840 (+ 937us) server_negotiation.cc:299] Negotiation successful
0620 07:08:49.035278 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":328,"threads_started":1}
I20260620 07:08:49.038569 897 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } }
I20260620 07:08:49.039175 897 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:49.039350 897 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc64646255d74aa98dcb134871780343, State: Initialized, Role: FOLLOWER
I20260620 07:08:49.040109 897 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } }
I20260620 07:08:49.040410 897 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:49.040560 897 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:49.040759 897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:49.042196 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:49.044270 897 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } }
I20260620 07:08:49.044912 897 leader_election.cc:302] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc64646255d74aa98dcb134871780343; no voters:
I20260620 07:08:49.045753 897 leader_election.cc:288] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:49.046124 902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:49.047124 902 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [term 1 LEADER]: Becoming Leader. State: Replica: fc64646255d74aa98dcb134871780343, State: Running, Role: LEADER
I20260620 07:08:49.047955 902 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } }
I20260620 07:08:49.048178 897 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:49.058796 906 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc64646255d74aa98dcb134871780343. Latest consensus state: current_term: 1 leader_uuid: "fc64646255d74aa98dcb134871780343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } } }
I20260620 07:08:49.060030 906 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:49.059605 905 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc64646255d74aa98dcb134871780343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc64646255d74aa98dcb134871780343" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35907 } } }
I20260620 07:08:49.060607 905 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:49.070626 916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:49.088148 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:49.067712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54499 (local address 127.11.81.254:35907)
0620 07:08:49.067931 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:49.067937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:49.069010 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 07:08:49.069284 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:49.069289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:49.069377 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:49.069479 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:49.071592 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:49.072834 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:49.074759 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:49.075503 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:49.075615 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:49.080486 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:49.080494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:49.080499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:49.080556 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:49.084540 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:49.085685 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:49.085692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:49.085696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:49.085860 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:49.086403 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:49.086407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:49.086409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:49.086736 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:49.086986 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:49.087564 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 07:08:49.087760 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:08:49.090583 916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:49.099915 916 catalog_manager.cc:1347] Generated new cluster ID: 902fcb9662754ac2a4b4f1c64cadfbe3
I20260620 07:08:49.100047 916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:49.114776 916 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:49.116699 916 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:49.131943 916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc64646255d74aa98dcb134871780343: Generated new TSK 0
I20260620 07:08:49.133072 916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:49.138556 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 830
I20260620 07:08:49.140149 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:49.239092 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 830
W20260620 07:08:49.253367 815 connection.cc:535] client connection to 127.11.81.254:35907 recv error: Network error: failed to read from TLS socket (remote: 127.11.81.254:35907): Connection reset by peer (error 104)
2026-06-20T07:08:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:49 dist-test-slave-wvz3 krb5kdc[929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:49 dist-test-slave-wvz3 krb5kdc[929](info): set up 2 sockets
Jun 20 07:08:49 dist-test-slave-wvz3 krb5kdc[929](info): commencing operation
krb5kdc: starting...
W20260620 07:08:50.263815 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:50 dist-test-slave-wvz3 krb5kdc[929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:50.522153 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:32771
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:32771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:50.849591 942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:50.849916 942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:50.850054 942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:50.857529 942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:50.857671 942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:50.857729 942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:50.857816 942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:50.857879 942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:50.867969 942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:32771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:50.869285 942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:50.870579 942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:50.882323 950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:50.882292 947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:50.882359 948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:50.883091 942 server_base.cc:1034] running on GCE node
I20260620 07:08:50.883903 942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:50.885324 942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:50.886530 942 hybrid_clock.cc:648] HybridClock initialized: now 1781939330886477 us; error 43 us; skew 500 ppm
I20260620 07:08:50.886937 942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:50 dist-test-slave-wvz3 krb5kdc[929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939330, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:50.890014 942 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:50.891525 942 webserver.cc:460] Webserver started at http://127.11.81.254:46583/ using document root <none> and password file <none>
I20260620 07:08:50.892191 942 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:50.892295 942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:50.892537 942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:50.895506 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "c3f1d512004f48638475488c0686ee06"
format_stamp: "Formatted at 2026-06-20 07:08:50 on dist-test-slave-wvz3"
server_key: "746c23495342f32c7d99ce6812df5714"
server_key_iv: "963b6bacfa399505336f73e8353cd1c5"
server_key_version: "encryptionkey@0"
I20260620 07:08:50.896284 942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "c3f1d512004f48638475488c0686ee06"
format_stamp: "Formatted at 2026-06-20 07:08:50 on dist-test-slave-wvz3"
server_key: "746c23495342f32c7d99ce6812df5714"
server_key_iv: "963b6bacfa399505336f73e8353cd1c5"
server_key_version: "encryptionkey@0"
I20260620 07:08:50.901407 942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 07:08:50.904958 957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:50.906431 942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:08:50.906587 942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "c3f1d512004f48638475488c0686ee06"
format_stamp: "Formatted at 2026-06-20 07:08:50 on dist-test-slave-wvz3"
server_key: "746c23495342f32c7d99ce6812df5714"
server_key_iv: "963b6bacfa399505336f73e8353cd1c5"
server_key_version: "encryptionkey@0"
I20260620 07:08:50.906764 942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:50.918270 942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:50.922634 942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:50.922881 942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:50.941061 942 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:32771
I20260620 07:08:50.941093 1008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:32771 every 8 connection(s)
I20260620 07:08:50.942490 942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:50.946056 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 942
I20260620 07:08:50.946215 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:50.946364 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:50.946688 11591 external_mini_cluster.cc:1428] Setting key 5e4609637968d90657b3e44238f57d3e
I20260620 07:08:50.960894 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: Bootstrap starting.
I20260620 07:08:50.961229 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:50.948623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48255 (local address 127.11.81.254:32771)
0620 07:08:50.949509 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:50.949521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:50.949600 (+ 79us) server_negotiation.cc:407] Connection header received
0620 07:08:50.951231 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:50.951256 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:50.951789 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:50.952197 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:50.952976 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:50.954427 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:50.956330 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:50.957190 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:50.957378 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:50.958755 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:50.958764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:50.958781 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:50.959421 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:50.959510 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:50.959542 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:50.959695 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:50.959948 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 07:08:50.960138 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":369,"threads_started":1}
I20260620 07:08:50.964246 1009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:50.965034 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:50.965405 1009 log.cc:824] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:50.968353 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: No bootstrap required, opened a new log
I20260620 07:08:50.976117 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:50.969240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38583 (local address 127.11.81.254:32771)
0620 07:08:50.969442 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:50.969449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:50.969887 (+ 438us) server_negotiation.cc:407] Connection header received
0620 07:08:50.970086 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:50.970092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:50.970167 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:50.970252 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:50.970846 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:50.971694 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:50.973513 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:50.974009 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:50.974106 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:50.975386 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:50.975392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:50.975396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:50.975417 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:50.975459 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:50.975476 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:50.975559 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:50.975725 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:08:50.975771 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:08:50.978063 1009 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } }
I20260620 07:08:50.978459 1009 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:50.978551 1009 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f1d512004f48638475488c0686ee06, State: Initialized, Role: FOLLOWER
I20260620 07:08:50.979100 1009 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } }
I20260620 07:08:50.979383 1009 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:50.979508 1009 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:50.979662 1009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:50.981956 1009 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } }
I20260620 07:08:50.982471 1009 leader_election.cc:302] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f1d512004f48638475488c0686ee06; no voters:
I20260620 07:08:50.983078 1009 leader_election.cc:288] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:50.983201 1019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:50.985206 1009 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:50.987496 1019 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [term 1 LEADER]: Becoming Leader. State: Replica: c3f1d512004f48638475488c0686ee06, State: Running, Role: LEADER
I20260620 07:08:50.988054 1019 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } }
I20260620 07:08:50.990954 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f1d512004f48638475488c0686ee06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } } }
I20260620 07:08:50.991387 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:50.992374 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f1d512004f48638475488c0686ee06. Latest consensus state: current_term: 1 leader_uuid: "c3f1d512004f48638475488c0686ee06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f1d512004f48638475488c0686ee06" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 32771 } } }
I20260620 07:08:50.992693 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06 [sys.catalog]: This master's current role is: LEADER
W20260620 07:08:51.000023 1033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:08:51.000152 1033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:08:51.000316 1034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:51.002578 1034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:51.012421 1034 catalog_manager.cc:1347] Generated new cluster ID: 9dfd64e1f9e947e2879a2ad74c606381
I20260620 07:08:51.012558 1034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:51.037031 1034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:51.038955 1034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:51.046468 1034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f1d512004f48638475488c0686ee06: Generated new TSK 0
I20260620 07:08:51.047596 1034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:51.059228 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 942
W20260620 07:08:51.060530 1004 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48255
W20260620 07:08:51.061328 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 942: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:51.061493 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 942
W20260620 07:08:51.075582 927 connection.cc:535] client connection to 127.11.81.254:32771 recv error: Network error: failed to read from TLS socket (remote: 127.11.81.254:32771): Connection reset by peer (error 104)
2026-06-20T07:08:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:51 dist-test-slave-wvz3 krb5kdc[1040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:51 dist-test-slave-wvz3 krb5kdc[1040](info): set up 2 sockets
Jun 20 07:08:51 dist-test-slave-wvz3 krb5kdc[1040](info): commencing operation
krb5kdc: starting...
W20260620 07:08:52.087776 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:52 dist-test-slave-wvz3 krb5kdc[1040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:52.346674 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41083
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:34317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:41083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:52.677642 1053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:52.677973 1053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:52.678068 1053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:52.685490 1053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:52.685618 1053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:52.685664 1053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:52.685725 1053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:52.685770 1053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:52.696028 1053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:34317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:41083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:52.697299 1053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:52.698599 1053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:52.711028 1059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:52.711028 1058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:52.711040 1061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:52.711457 1053 server_base.cc:1034] running on GCE node
I20260620 07:08:52.712291 1053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:52.713755 1053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:52.714996 1053 hybrid_clock.cc:648] HybridClock initialized: now 1781939332714935 us; error 49 us; skew 500 ppm
I20260620 07:08:52.715410 1053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:52 dist-test-slave-wvz3 krb5kdc[1040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939332, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:52.718616 1053 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:52.720177 1053 webserver.cc:460] Webserver started at http://127.11.81.254:42273/ using document root <none> and password file <none>
I20260620 07:08:52.721094 1053 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:52.721233 1053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:52.721606 1053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:52.725060 1053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "1699f3f021824493a6b29f945bc638a2"
format_stamp: "Formatted at 2026-06-20 07:08:52 on dist-test-slave-wvz3"
server_key: "d6aa10111808ef202091389d2c8d71d6"
server_key_iv: "2cc2fb0f1cc59fd55fb48674c8b72dde"
server_key_version: "encryptionkey@0"
I20260620 07:08:52.725850 1053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "1699f3f021824493a6b29f945bc638a2"
format_stamp: "Formatted at 2026-06-20 07:08:52 on dist-test-slave-wvz3"
server_key: "d6aa10111808ef202091389d2c8d71d6"
server_key_iv: "2cc2fb0f1cc59fd55fb48674c8b72dde"
server_key_version: "encryptionkey@0"
I20260620 07:08:52.731336 1053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:08:52.735069 1068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:52.736696 1053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:08:52.736873 1053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "1699f3f021824493a6b29f945bc638a2"
format_stamp: "Formatted at 2026-06-20 07:08:52 on dist-test-slave-wvz3"
server_key: "d6aa10111808ef202091389d2c8d71d6"
server_key_iv: "2cc2fb0f1cc59fd55fb48674c8b72dde"
server_key_version: "encryptionkey@0"
I20260620 07:08:52.737079 1053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:52.743017 1053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:52.747457 1053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:52.747746 1053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:52.765789 1053 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:41083
I20260620 07:08:52.765869 1119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:41083 every 8 connection(s)
I20260620 07:08:52.767481 1053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:52.771437 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:52.771926 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1053
I20260620 07:08:52.772114 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:52.772653 11591 external_mini_cluster.cc:1428] Setting key fc803a3b3222c50a0abb12b706a75bfc
I20260620 07:08:52.781455 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:52.774471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45771 (local address 127.11.81.254:41083)
0620 07:08:52.775461 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:52.775473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:52.775514 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:08:52.777886 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:52.777917 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:52.778356 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:52.778782 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:52.779847 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:52.779853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:52.779867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:52.779880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:52.779935 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:52.779957 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:52.780135 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:52.780251 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:08:52.780425 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":481,"threads_started":1}
I20260620 07:08:52.785089 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2: Bootstrap starting.
I20260620 07:08:52.785259 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:52.789053 1120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:52.790354 1120 log.cc:824] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:52.792833 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:52.790425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46203 (local address 127.11.81.254:41083)
0620 07:08:52.790635 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:52.790642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:52.791049 (+ 407us) server_negotiation.cc:407] Connection header received
0620 07:08:52.791212 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:52.791217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:52.791276 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:52.791421 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:52.792244 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:52.792248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:52.792251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:52.792265 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:52.792302 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:52.792316 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:52.792402 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:52.792514 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 07:08:52.792575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:08:52.793673 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2: No bootstrap required, opened a new log
I20260620 07:08:52.801141 1120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } }
I20260620 07:08:52.801550 1120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:52.801643 1120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1699f3f021824493a6b29f945bc638a2, State: Initialized, Role: FOLLOWER
I20260620 07:08:52.802292 1120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } }
I20260620 07:08:52.802522 1120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:52.802649 1120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:52.802800 1120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:52.805038 1120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } }
I20260620 07:08:52.805538 1120 leader_election.cc:302] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1699f3f021824493a6b29f945bc638a2; no voters:
I20260620 07:08:52.806165 1120 leader_election.cc:288] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:52.806299 1130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:52.807121 1130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1699f3f021824493a6b29f945bc638a2, State: Running, Role: LEADER
I20260620 07:08:52.807650 1130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } }
I20260620 07:08:52.808286 1120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:52.810321 1131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1699f3f021824493a6b29f945bc638a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } } }
I20260620 07:08:52.810333 1132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1699f3f021824493a6b29f945bc638a2. Latest consensus state: current_term: 1 leader_uuid: "1699f3f021824493a6b29f945bc638a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1699f3f021824493a6b29f945bc638a2" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 41083 } } }
I20260620 07:08:52.810810 1131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:52.810810 1132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:52.813256 1137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:52.819433 1137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:52.829419 1137 catalog_manager.cc:1347] Generated new cluster ID: 9de3595ce19e4d6e82635154e4dcc6a4
I20260620 07:08:52.829612 1137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:52.885429 1137 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:52.885905 1137 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:52.892937 1137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1699f3f021824493a6b29f945bc638a2: Generated new TSK 0
I20260620 07:08:52.894097 1137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:52.910696 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1053
W20260620 07:08:52.911911 1115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45771
W20260620 07:08:52.912648 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:52.912813 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1053
2026-06-20T07:08:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:53 dist-test-slave-wvz3 krb5kdc[1151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:53 dist-test-slave-wvz3 krb5kdc[1151](info): set up 2 sockets
Jun 20 07:08:53 dist-test-slave-wvz3 krb5kdc[1151](info): commencing operation
krb5kdc: starting...
W20260620 07:08:53.947598 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:54 dist-test-slave-wvz3 krb5kdc[1151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:54.206632 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:42395
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:33493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:54.553023 1164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:54.553333 1164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:54.553440 1164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:54.561303 1164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:54.561496 1164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:54.561560 1164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:54.561622 1164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:54.561666 1164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:54.572048 1164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:33493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:54.573323 1164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:54.574659 1164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:54.586799 1170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:54.587000 1172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:54.587218 1169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:54.587728 1164 server_base.cc:1034] running on GCE node
I20260620 07:08:54.588455 1164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:54.589938 1164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:54.591164 1164 hybrid_clock.cc:648] HybridClock initialized: now 1781939334591102 us; error 46 us; skew 500 ppm
I20260620 07:08:54.591547 1164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:54 dist-test-slave-wvz3 krb5kdc[1151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939334, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:54.599711 1164 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:54.601258 1164 webserver.cc:460] Webserver started at http://127.11.81.254:41877/ using document root <none> and password file <none>
I20260620 07:08:54.601959 1164 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:54.602079 1164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:54.602376 1164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:54.605614 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "62f9426826b94e0ea9823d990570abde"
format_stamp: "Formatted at 2026-06-20 07:08:54 on dist-test-slave-wvz3"
server_key: "3e2c4e0a338818cf3002a0f18ee0d73e"
server_key_iv: "f9a47ef489bbd014e27bd8b0cb84d78a"
server_key_version: "encryptionkey@0"
I20260620 07:08:54.606396 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "62f9426826b94e0ea9823d990570abde"
format_stamp: "Formatted at 2026-06-20 07:08:54 on dist-test-slave-wvz3"
server_key: "3e2c4e0a338818cf3002a0f18ee0d73e"
server_key_iv: "f9a47ef489bbd014e27bd8b0cb84d78a"
server_key_version: "encryptionkey@0"
I20260620 07:08:54.611953 1164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 07:08:54.615927 1179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:54.617537 1164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:54.617712 1164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "62f9426826b94e0ea9823d990570abde"
format_stamp: "Formatted at 2026-06-20 07:08:54 on dist-test-slave-wvz3"
server_key: "3e2c4e0a338818cf3002a0f18ee0d73e"
server_key_iv: "f9a47ef489bbd014e27bd8b0cb84d78a"
server_key_version: "encryptionkey@0"
I20260620 07:08:54.617918 1164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:54.642031 1164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:54.646524 1164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:54.646811 1164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:54.665264 1164 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:42395
I20260620 07:08:54.665299 1230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:42395 every 8 connection(s)
I20260620 07:08:54.666641 1164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:54.670499 1231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:54.673764 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1164
I20260620 07:08:54.673943 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:54.674467 11591 external_mini_cluster.cc:1428] Setting key 1406642019a232e51a288adba4cafd14
I20260620 07:08:54.683112 1231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: Bootstrap starting.
I20260620 07:08:54.687502 1231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:54.688930 1231 log.cc:824] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:08:54 dist-test-slave-wvz3 krb5kdc[1151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:08:54.692377 1231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: No bootstrap required, opened a new log
I20260620 07:08:54.698583 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:54.676440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36705 (local address 127.11.81.254:42395)
0620 07:08:54.677240 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:54.677253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:54.677298 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:08:54.678987 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:54.679013 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:54.679558 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:54.679926 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:54.681642 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:54.682952 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:54.684784 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:54.685517 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:54.685655 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:54.690898 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:54.690907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:54.690923 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:54.690962 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:54.694166 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:54.694970 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:54.694978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:54.694985 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:54.695176 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:54.695607 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:54.695612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:54.695614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:54.696191 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:54.696513 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:54.696975 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 07:08:54.697441 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":351,"threads_started":1}
I20260620 07:08:54.700675 1231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } }
I20260620 07:08:54.701078 1231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:54.701241 1231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f9426826b94e0ea9823d990570abde, State: Initialized, Role: FOLLOWER
I20260620 07:08:54.701886 1231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } }
I20260620 07:08:54.702129 1231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:54.702267 1231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:54.702440 1231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:54.702514 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:54.704944 1231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } }
I20260620 07:08:54.705444 1231 leader_election.cc:302] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f9426826b94e0ea9823d990570abde; no voters:
I20260620 07:08:54.706086 1231 leader_election.cc:288] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:54.706194 1240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:54.708153 1240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [term 1 LEADER]: Becoming Leader. State: Replica: 62f9426826b94e0ea9823d990570abde, State: Running, Role: LEADER
I20260620 07:08:54.708595 1231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:54.708842 1240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } }
I20260620 07:08:54.711845 1241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f9426826b94e0ea9823d990570abde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } } }
I20260620 07:08:54.711931 1242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f9426826b94e0ea9823d990570abde. Latest consensus state: current_term: 1 leader_uuid: "62f9426826b94e0ea9823d990570abde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f9426826b94e0ea9823d990570abde" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 42395 } } }
I20260620 07:08:54.712260 1241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:54.712281 1242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde [sys.catalog]: This master's current role is: LEADER
W20260620 07:08:54.721725 1257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:08:54.721865 1257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:08:54.722038 1250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:54.723896 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:54.707070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40199 (local address 127.11.81.254:42395)
0620 07:08:54.707245 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:54.707252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:54.708139 (+ 887us) server_negotiation.cc:407] Connection header received
0620 07:08:54.708493 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:54.708499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:54.708583 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:54.708672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:54.710249 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:54.711401 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:54.713231 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:54.713964 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:54.714090 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:54.717687 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:54.717692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:54.717695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:08:54.717728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:54.720724 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:54.721495 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:54.721501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:54.721504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:54.721637 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:08:54.722071 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:08:54.722080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:08:54.722083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:08:54.722597 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:54.722866 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:54.723246 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 07:08:54.723531 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:08:54.724341 1250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:54.734759 1250 catalog_manager.cc:1347] Generated new cluster ID: c4e14084cce743698be266ea93d8d9fc
I20260620 07:08:54.734894 1250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:54.767176 1250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:54.769093 1250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:54.797387 1250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f9426826b94e0ea9823d990570abde: Generated new TSK 0
I20260620 07:08:54.798543 1250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:54.813423 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1164
I20260620 07:08:54.814647 1226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:08:54.910895 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1164
2026-06-20T07:08:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:55 dist-test-slave-wvz3 krb5kdc[1263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:55 dist-test-slave-wvz3 krb5kdc[1263](info): set up 2 sockets
Jun 20 07:08:55 dist-test-slave-wvz3 krb5kdc[1263](info): commencing operation
krb5kdc: starting...
W20260620 07:08:55.959686 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:56 dist-test-slave-wvz3 krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:56.223723 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35051
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:56.561089 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:56.561417 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:56.561551 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:56.568969 1276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:56.569145 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:56.569231 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:56.569311 1276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:56.569373 1276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:56.579488 1276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:56.580828 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:56.582185 1276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:56.594210 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:56.594285 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:56.594326 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:56.594991 1276 server_base.cc:1034] running on GCE node
I20260620 07:08:56.595925 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:56.597414 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:56.598639 1276 hybrid_clock.cc:648] HybridClock initialized: now 1781939336598597 us; error 40 us; skew 500 ppm
I20260620 07:08:56.598990 1276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:56 dist-test-slave-wvz3 krb5kdc[1263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939336, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:56.602118 1276 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:56.603756 1276 webserver.cc:460] Webserver started at http://127.11.81.254:40493/ using document root <none> and password file <none>
I20260620 07:08:56.604426 1276 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:56.604557 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:56.604843 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:56.607929 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e"
format_stamp: "Formatted at 2026-06-20 07:08:56 on dist-test-slave-wvz3"
server_key: "3353ff0b1cceef0ae5e8bb4c9c906cfe"
server_key_iv: "cc2fb32e192b97676c8a965b79e528aa"
server_key_version: "encryptionkey@0"
I20260620 07:08:56.608744 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e"
format_stamp: "Formatted at 2026-06-20 07:08:56 on dist-test-slave-wvz3"
server_key: "3353ff0b1cceef0ae5e8bb4c9c906cfe"
server_key_iv: "cc2fb32e192b97676c8a965b79e528aa"
server_key_version: "encryptionkey@0"
I20260620 07:08:56.614218 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:08:56.618139 1291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:56.619701 1276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:08:56.619871 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e"
format_stamp: "Formatted at 2026-06-20 07:08:56 on dist-test-slave-wvz3"
server_key: "3353ff0b1cceef0ae5e8bb4c9c906cfe"
server_key_iv: "cc2fb32e192b97676c8a965b79e528aa"
server_key_version: "encryptionkey@0"
I20260620 07:08:56.620077 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:56.643438 1276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:56.648269 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:56.648572 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:56.666849 1276 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:35051
I20260620 07:08:56.666880 1342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:35051 every 8 connection(s)
I20260620 07:08:56.668352 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:56.670126 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1276
I20260620 07:08:56.670286 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:56.670709 11591 external_mini_cluster.cc:1428] Setting key 1979d52136e4c520cfc29166b6ba46d4
I20260620 07:08:56.672701 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:56.684512 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e: Bootstrap starting.
I20260620 07:08:56.684868 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:56.672567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45755 (local address 127.11.81.254:35051)
0620 07:08:56.673414 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:56.673427 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:56.673472 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:08:56.675244 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:56.675269 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:56.675763 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:56.676237 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:56.676860 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:56.678337 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:56.679859 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:56.680607 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:56.680801 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:56.682186 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:56.682195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:56.682217 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:56.682865 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:56.682951 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:56.682981 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:56.683097 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:56.683406 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 07:08:56.683696 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":371,"threads_started":1}
I20260620 07:08:56.688277 1343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:56.689038 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:56.689455 1343 log.cc:824] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:56.692446 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e: No bootstrap required, opened a new log
I20260620 07:08:56.700251 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:56.693377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35959 (local address 127.11.81.254:35051)
0620 07:08:56.693590 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:56.693596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:56.694013 (+ 417us) server_negotiation.cc:407] Connection header received
0620 07:08:56.694205 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:56.694209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:56.694274 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:56.694368 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:56.694907 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:56.695753 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:56.697638 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:08:56.698170 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:08:56.698270 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:08:56.699513 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:56.699517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:56.699521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:56.699542 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:56.699582 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:56.699598 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:56.699676 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:56.699815 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 07:08:56.699897 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:08:56.702064 1343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } }
I20260620 07:08:56.702482 1343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:56.702584 1343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a05b007e1ea49f8b8a1fac1dd1d3a2e, State: Initialized, Role: FOLLOWER
I20260620 07:08:56.703153 1343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } }
I20260620 07:08:56.703408 1343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:56.703534 1343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:56.703689 1343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:56.705889 1343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } }
I20260620 07:08:56.706396 1343 leader_election.cc:302] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a05b007e1ea49f8b8a1fac1dd1d3a2e; no voters:
I20260620 07:08:56.707043 1343 leader_election.cc:288] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:56.707162 1353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:56.707806 1353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [term 1 LEADER]: Becoming Leader. State: Replica: 0a05b007e1ea49f8b8a1fac1dd1d3a2e, State: Running, Role: LEADER
I20260620 07:08:56.708439 1353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } }
I20260620 07:08:56.709424 1343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:56.711416 1354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } } }
I20260620 07:08:56.711469 1355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a05b007e1ea49f8b8a1fac1dd1d3a2e. Latest consensus state: current_term: 1 leader_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05b007e1ea49f8b8a1fac1dd1d3a2e" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 35051 } } }
I20260620 07:08:56.711906 1354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:56.711907 1355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:56.712831 1360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:56.718623 1360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:56.728394 1360 catalog_manager.cc:1347] Generated new cluster ID: a5d7a31ccc5040e3b5916f84ca32a0d4
I20260620 07:08:56.728539 1360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:56.741847 1360 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:56.743251 1360 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:56.751658 1360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a05b007e1ea49f8b8a1fac1dd1d3a2e: Generated new TSK 0
I20260620 07:08:56.752867 1360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:56.767031 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1276
W20260620 07:08:56.768427 1338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45755
W20260620 07:08:56.769232 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:56.769382 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1276
2026-06-20T07:08:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:56 dist-test-slave-wvz3 krb5kdc[1374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:56 dist-test-slave-wvz3 krb5kdc[1374](info): set up 2 sockets
Jun 20 07:08:56 dist-test-slave-wvz3 krb5kdc[1374](info): commencing operation
krb5kdc: starting...
W20260620 07:08:57.795739 11591 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:57 dist-test-slave-wvz3 krb5kdc[1374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:58.054829 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43733
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:08:58.387508 1387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:08:58.387830 1387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:08:58.387962 1387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:08:58.395401 1387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:08:58.395543 1387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:08:58.395619 1387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:08:58.395704 1387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:08:58.395763 1387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:08:58.405805 1387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:08:58.407181 1387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:08:58.408520 1387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:08:58.420429 1392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:58.420464 1395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:08:58.420363 1393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:08:58.421233 1387 server_base.cc:1034] running on GCE node
I20260620 07:08:58.422168 1387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:08:58.423699 1387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:08:58.424939 1387 hybrid_clock.cc:648] HybridClock initialized: now 1781939338424880 us; error 40 us; skew 500 ppm
I20260620 07:08:58.425297 1387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:08:58 dist-test-slave-wvz3 krb5kdc[1374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939338, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:08:58.428546 1387 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:08:58.430191 1387 webserver.cc:460] Webserver started at http://127.11.81.254:40705/ using document root <none> and password file <none>
I20260620 07:08:58.430872 1387 fs_manager.cc:359] Metadata directory not provided
I20260620 07:08:58.430997 1387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:08:58.431347 1387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:08:58.434351 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "cd6142ae6a6343db83c102873500dcc1"
format_stamp: "Formatted at 2026-06-20 07:08:58 on dist-test-slave-wvz3"
server_key: "77ab5edb54377e53144b4c6f582380ec"
server_key_iv: "0323219c167c64ffef2658429bae76ce"
server_key_version: "encryptionkey@0"
I20260620 07:08:58.435101 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "cd6142ae6a6343db83c102873500dcc1"
format_stamp: "Formatted at 2026-06-20 07:08:58 on dist-test-slave-wvz3"
server_key: "77ab5edb54377e53144b4c6f582380ec"
server_key_iv: "0323219c167c64ffef2658429bae76ce"
server_key_version: "encryptionkey@0"
I20260620 07:08:58.440551 1387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:08:58.444272 1402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:08:58.445848 1387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:08:58.446056 1387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "cd6142ae6a6343db83c102873500dcc1"
format_stamp: "Formatted at 2026-06-20 07:08:58 on dist-test-slave-wvz3"
server_key: "77ab5edb54377e53144b4c6f582380ec"
server_key_iv: "0323219c167c64ffef2658429bae76ce"
server_key_version: "encryptionkey@0"
I20260620 07:08:58.446297 1387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:08:58.452629 1387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:08:58.457086 1387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:08:58.457406 1387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:08:58.476240 1387 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:43733
I20260620 07:08:58.476289 1453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:43733 every 8 connection(s)
I20260620 07:08:58.477699 1387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:08:58.480190 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1387
I20260620 07:08:58.480373 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:08:58.480904 11591 external_mini_cluster.cc:1428] Setting key 5d8174f17e1d54793e6166457209aac6
I20260620 07:08:58.482491 1454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:08:58.489033 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:58.482848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43523 (local address 127.11.81.254:43733)
0620 07:08:58.483650 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:58.483664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:58.483705 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:08:58.485301 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:58.485325 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:58.485699 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:58.486049 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:58.487191 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:58.487198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:58.487211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:58.487227 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:58.487418 (+ 191us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:58.487456 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:58.487660 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:58.487759 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:08:58.487971 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":362,"threads_started":1}
I20260620 07:08:58.492808 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:08:58.495716 1454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1: Bootstrap starting.
I20260620 07:08:58.499800 1454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1: Neither blocks nor log segments found. Creating new log.
I20260620 07:08:58.500638 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:08:58.498108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54065 (local address 127.11.81.254:43733)
0620 07:08:58.498298 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:08:58.498303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:08:58.498577 (+ 274us) server_negotiation.cc:407] Connection header received
0620 07:08:58.498802 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:08:58.498806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:08:58.498857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:08:58.498941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:08:58.499940 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:08:58.499946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:08:58.499950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:08:58.499969 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:08:58.500015 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:08:58.500031 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:08:58.500133 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:08:58.500285 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 07:08:58.500336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:08:58.501232 1454 log.cc:824] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:08:58.504477 1454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1: No bootstrap required, opened a new log
I20260620 07:08:58.512218 1454 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } }
I20260620 07:08:58.512809 1454 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:08:58.512921 1454 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6142ae6a6343db83c102873500dcc1, State: Initialized, Role: FOLLOWER
I20260620 07:08:58.513546 1454 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } }
I20260620 07:08:58.513782 1454 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:08:58.513924 1454 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:08:58.514087 1454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:08:58.516400 1454 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } }
I20260620 07:08:58.516956 1454 leader_election.cc:302] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd6142ae6a6343db83c102873500dcc1; no voters:
I20260620 07:08:58.517642 1454 leader_election.cc:288] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:08:58.517807 1464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:08:58.518493 1464 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [term 1 LEADER]: Becoming Leader. State: Replica: cd6142ae6a6343db83c102873500dcc1, State: Running, Role: LEADER
I20260620 07:08:58.519078 1464 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } }
I20260620 07:08:58.519905 1454 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:08:58.521839 1465 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd6142ae6a6343db83c102873500dcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } } }
I20260620 07:08:58.522502 1465 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:58.525013 1466 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6142ae6a6343db83c102873500dcc1. Latest consensus state: current_term: 1 leader_uuid: "cd6142ae6a6343db83c102873500dcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6142ae6a6343db83c102873500dcc1" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 43733 } } }
I20260620 07:08:58.525288 1466 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:08:58.525297 1471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:08:58.530336 1471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:08:58.540518 1471 catalog_manager.cc:1347] Generated new cluster ID: 7a3807e32824421d9d7dcf62773a4317
I20260620 07:08:58.540663 1471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:08:58.556919 1471 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:08:58.557477 1471 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:08:58.568642 1471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd6142ae6a6343db83c102873500dcc1: Generated new TSK 0
I20260620 07:08:58.569859 1471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:08:58.578462 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1387
W20260620 07:08:58.579644 1449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43523
W20260620 07:08:58.580395 11591 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:08:58.580546 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1387
2026-06-20T07:08:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1811 ms)
[----------] 18 tests from AuthTokenIssuingTest (33836 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:08:58 dist-test-slave-wvz3 krb5kdc[1485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:08:58 dist-test-slave-wvz3 krb5kdc[1485](info): set up 2 sockets
Jun 20 07:08:58 dist-test-slave-wvz3 krb5kdc[1485](info): commencing operation
krb5kdc: starting...
W20260620 07:08:59.620194 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:08:59 dist-test-slave-wvz3 krb5kdc[1485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:08:59.887480 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40631
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:00.253618 1498 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:00.253993 1498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:00.254098 1498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:00.254222 1498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:00.261927 1498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:00.262050 1498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:00.262138 1498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:00.262210 1498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:00.262298 1498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:00.272686 1498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:00.274073 1498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:00.275561 1498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:00.288506 1504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:00.288538 1503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:00.288589 1506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:00.289587 1498 server_base.cc:1034] running on GCE node
I20260620 07:09:00.290489 1498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:00.292160 1498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:00.293565 1498 hybrid_clock.cc:648] HybridClock initialized: now 1781939340293470 us; error 51 us; skew 500 ppm
I20260620 07:09:00.293947 1498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:00 dist-test-slave-wvz3 krb5kdc[1485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:00.301244 1498 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:09:00.303036 1498 webserver.cc:460] Webserver started at http://127.11.81.254:33471/ using document root <none> and password file <none>
I20260620 07:09:00.303884 1498 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:00.304006 1498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:00.304332 1498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:00.307582 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "0b498dd542224a3399f1dbd7fc3dd2e4"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "71297f5d458ca3074338e9d669e9f131"
server_key_iv: "02d0764c33460da7f63ccc57fbcb0f13"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.308527 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "0b498dd542224a3399f1dbd7fc3dd2e4"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "71297f5d458ca3074338e9d669e9f131"
server_key_iv: "02d0764c33460da7f63ccc57fbcb0f13"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.314308 1498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:00.318358 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:00.319967 1498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:00.320165 1498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "0b498dd542224a3399f1dbd7fc3dd2e4"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "71297f5d458ca3074338e9d669e9f131"
server_key_iv: "02d0764c33460da7f63ccc57fbcb0f13"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.320487 1498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:00.344597 1498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:00.349706 1498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:00.350016 1498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:00.368896 1498 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:40631
I20260620 07:09:00.368978 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:40631 every 8 connection(s)
I20260620 07:09:00.370303 1498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:00.374361 1565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:00.375651 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1498
I20260620 07:09:00.375856 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:09:00.376543 11591 external_mini_cluster.cc:1428] Setting key 5b0355776fa6892d6912c3fc43c3db1b
I20260620 07:09:00.381479 1565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:00 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.253@KRBTEST.COM" created.
I20260620 07:09:00.406198 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.384909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40631 (local address 127.0.0.1:36978)
0620 07:09:00.385811 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.385871 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:00.387789 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:00.388832 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:00.388837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:00.389531 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:00.389939 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.389959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.392215 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.392219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.393409 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.393427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.394691 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.394697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.394839 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.395604 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:00.395633 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:00.398160 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:00.401918 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.401930 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.401940 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.402413 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.402934 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.402938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.402940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.403146 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.404119 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:00.404126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:00.404320 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 07:09:00.404601 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 07:09:00.405037 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":374,"threads_started":1}
I20260620 07:09:00.406231 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.384909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.11.81.254:40631)
0620 07:09:00.385783 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:00.385797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:00.385999 (+ 202us) server_negotiation.cc:407] Connection header received
0620 07:09:00.388256 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:00.388264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:00.388578 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:00.388790 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:00.390171 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.392041 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.393601 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.394453 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.394619 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.398508 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:00.398517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:00.398532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:00.398573 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:00.401660 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.402605 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.402612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.402615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.402782 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.403323 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.403328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.403331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.403960 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:00.404180 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:00.404787 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 07:09:00.405228 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":677,"spinlock_wait_cycles":14080,"thread_start_us":456,"threads_started":1}
I20260620 07:09:00.410254 1565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:00.411579 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.411055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:58960)
0620 07:09:00.411300 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.411444 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:09:00.411878 1514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
I20260620 07:09:00.413565 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.413144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:58976)
0620 07:09:00.413329 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.413415 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 07:09:00.413832 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:32795: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
I20260620 07:09:00.438570 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:32795
--webserver_interface=127.11.81.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:09:00.463219 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } attempt: 1
I20260620 07:09:00.464596 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.464062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:58986)
0620 07:09:00.464278 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.464454 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 07:09:00.465641 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.465265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:58994)
0620 07:09:00.465438 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.465510 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 07:09:00.465837 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:32795: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
I20260620 07:09:00.534139 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } attempt: 2
I20260620 07:09:00.535485 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.534869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59010)
0620 07:09:00.535133 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.535235 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 07:09:00.536651 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.536238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59018)
0620 07:09:00.536416 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.536511 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 07:09:00.536862 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:32795: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
I20260620 07:09:00.628118 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } attempt: 3
I20260620 07:09:00.629258 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.628820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59028)
0620 07:09:00.629010 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.629107 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:09:00.630476 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.630047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59042)
0620 07:09:00.630229 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.630305 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 07:09:00.630693 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:32795: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
I20260620 07:09:00.773959 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } attempt: 4
I20260620 07:09:00.775162 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.774613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59044)
0620 07:09:00.774902 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.775012 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 07:09:00.776275 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.775901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59058)
0620 07:09:00.776058 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.776137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 07:09:00.776448 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:32795: Network error: Client connection negotiation failed: client connection to 127.11.81.253:32795: connect: Connection refused (error 111)
W20260620 07:09:00.777756 1572 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:00.778110 1572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:00.778213 1572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:00.778331 1572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:00.786026 1572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:00.786145 1572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:00.786228 1572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:00.786301 1572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:00.786391 1572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:00.797020 1572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:32795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.81.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:00.798511 1572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:00.799980 1572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:00.812994 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:00.813036 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:00.813073 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:00.814148 1572 server_base.cc:1034] running on GCE node
I20260620 07:09:00.815021 1572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:00.816615 1572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:00.817853 1572 hybrid_clock.cc:648] HybridClock initialized: now 1781939340817784 us; error 57 us; skew 500 ppm
I20260620 07:09:00.818217 1572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:00 dist-test-slave-wvz3 krb5kdc[1485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:00.821604 1572 init.cc:377] Logged in from keytab as kudu/127.11.81.253@KRBTEST.COM (short username kudu)
I20260620 07:09:00.823273 1572 webserver.cc:460] Webserver started at http://127.11.81.253:43075/ using document root <none> and password file <none>
I20260620 07:09:00.824054 1572 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:00.824144 1572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:00.824482 1572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:00.827688 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data/instance:
uuid: "d98238f707c848628a395bb7a78e4e17"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "07022059e947729d84246516ff6df807"
server_key_iv: "81fbf3af98e8c7f24b3b12805ce00fd3"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.828529 1572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal/instance:
uuid: "d98238f707c848628a395bb7a78e4e17"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "07022059e947729d84246516ff6df807"
server_key_iv: "81fbf3af98e8c7f24b3b12805ce00fd3"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.834163 1572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:00.838083 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:00.839666 1572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:00.839839 1572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "d98238f707c848628a395bb7a78e4e17"
format_stamp: "Formatted at 2026-06-20 07:09:00 on dist-test-slave-wvz3"
server_key: "07022059e947729d84246516ff6df807"
server_key_iv: "81fbf3af98e8c7f24b3b12805ce00fd3"
server_key_version: "encryptionkey@0"
I20260620 07:09:00.840075 1572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:00.860358 1572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:00.864902 1572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:00.865175 1572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:00.883119 1572 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.253:32795
I20260620 07:09:00.883175 1638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.253:32795 every 8 connection(s)
I20260620 07:09:00.884509 1572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
I20260620 07:09:00.885890 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1572
I20260620 07:09:00.886070 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-1/wal/instance
I20260620 07:09:00.886610 11591 external_mini_cluster.cc:1428] Setting key 2d280a73c36d58b7ae0e4f3cd547d22d
I20260620 07:09:00.889358 1639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:00.898233 1639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.81.252@KRBTEST.COM; defaulting to no policy
Jun 20 07:09:00 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.252@KRBTEST.COM" created.
I20260620 07:09:00.920475 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.901065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.11.81.254:40631)
0620 07:09:00.901306 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:00.901313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:00.902646 (+ 1333us) server_negotiation.cc:407] Connection header received
0620 07:09:00.904666 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:00.904672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:00.904768 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:00.904906 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:00.906476 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.907545 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.909371 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.910065 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.910228 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.914139 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:00.914144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:00.914148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:00.914186 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:00.917111 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.917912 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.917916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.917919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.918037 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.918515 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.918521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.918524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.918844 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:00.918966 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:00.919962 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 07:09:00.920198 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:09:00.921721 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.901570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40631 (local address 127.0.0.1:36994)
0620 07:09:00.902437 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.902492 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:00.904368 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:00.905127 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:00.905135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:00.905665 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:00.906258 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.906276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.907751 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.907756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.909179 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.909192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.910383 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.910389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.910609 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.911352 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:00.911385 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:00.913825 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:00.917276 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.917287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.917312 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.917768 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.918151 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.918156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.918160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.918372 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.918982 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:00.918990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:00.919498 (+ 508us) client_negotiation.cc:769] Sending connection context
0620 07:09:00.919811 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 07:09:00.920338 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
I20260620 07:09:00.923911 1639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:00 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:00.939778 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.924903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59060)
0620 07:09:00.925212 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.925232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:00.925365 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:00.926360 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:00.926364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:00.926849 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:00.926995 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.927004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.928513 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.928519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.929402 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:00.929414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.930280 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.930286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:00.930468 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.931179 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:00.931199 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:00.933591 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:00.937190 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.937195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.937200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.937582 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.937974 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:00.937980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:00.937983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:00.938123 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:00.938752 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:00.938756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:00.938906 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 07:09:00.939026 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:09:00.939370 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 07:09:00.939885 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.925131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.11.81.253:32795)
0620 07:09:00.925879 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:00.925886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:00.925913 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:00.925963 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:00.925970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:00.926180 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:00.926329 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:00.927176 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.928362 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.929549 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:00.930127 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:00.930242 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:00.933847 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:00.933876 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:00.933898 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:00.933939 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:00.937020 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.937724 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.937731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.937733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.937857 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:00.938232 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:00.938236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:00.938238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:00.938621 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:00.938825 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:00.939195 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 07:09:00.939505 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":590,"thread_start_us":438,"threads_started":1}
I20260620 07:09:00.942677 1639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:00.944025 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.943511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38134)
0620 07:09:00.943713 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.943835 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 07:09:00.944386 1590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:00.946131 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.945754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38146)
0620 07:09:00.945894 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.946006 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:09:00.946445 1639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
I20260620 07:09:00.951267 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:45061
--webserver_interface=127.11.81.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:09:00.995810 1639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 1
I20260620 07:09:00.997187 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.996608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38158)
0620 07:09:00.996845 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.996984 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 07:09:00.998333 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:00.997898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38166)
0620 07:09:00.998092 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:00.998190 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 07:09:00.998605 1639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:01.066898 1639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 2
I20260620 07:09:01.068154 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.067617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38168)
0620 07:09:01.067838 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.067962 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 07:09:01.069381 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.068944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38184)
0620 07:09:01.069120 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.069217 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 07:09:01.069643 1639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:01.075702 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } attempt: 5
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:01.091630 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.076504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59070)
0620 07:09:01.076694 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.076715 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.076823 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.077244 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.077248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.077663 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.077872 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.077882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.079506 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.079512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.080405 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.080421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.081594 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.081599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.081795 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.082449 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.082469 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.084682 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.088713 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.088718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.088721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.089191 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.089761 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.089766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.089769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.089968 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.090686 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.090690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.090889 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.091028 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.091338 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 07:09:01.091745 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.076559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.11.81.253:32795)
0620 07:09:01.076754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.076760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.076785 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:09:01.076960 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.076965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.077046 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.077170 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.078051 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.079254 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.080607 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.081373 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.081532 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.084936 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.084953 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.084958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.085005 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.088510 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.089336 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.089342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.089345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.089584 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.090093 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.090099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.090102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.090528 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.090693 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.091173 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.091446 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:09:01.092952 1565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:01.094105 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.093706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38186)
0620 07:09:01.093890 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.093968 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:09:01.095037 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.094706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38200)
0620 07:09:01.094860 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.094925 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:09:01.095225 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:01.146494 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 1
I20260620 07:09:01.147816 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.147217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38214)
0620 07:09:01.147523 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.147621 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 07:09:01.148893 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.148515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38224)
0620 07:09:01.148689 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.148766 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 07:09:01.149169 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:01.160933 1639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 3
I20260620 07:09:01.162340 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.161708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38236)
0620 07:09:01.161974 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.162109 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 07:09:01.163861 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.163239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38250)
0620 07:09:01.163581 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.163688 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
W20260620 07:09:01.164134 1639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
I20260620 07:09:01.217428 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 2
I20260620 07:09:01.218602 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.218115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38256)
0620 07:09:01.218332 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.218434 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:09:01.219893 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.219488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38260)
0620 07:09:01.219672 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.219757 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 07:09:01.220172 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
W20260620 07:09:01.302335 1646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:01.302706 1646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:01.302778 1646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:01.302865 1646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:09:01.307394 1639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 4
I20260620 07:09:01.308826 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.308241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38268)
0620 07:09:01.308507 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.308620 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:09:01.309988 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.309589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38272)
0620 07:09:01.309743 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.309839 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:09:01.310269 1639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
W20260620 07:09:01.311337 1646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:01.311430 1646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:01.311506 1646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:01.311570 1646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:01.311654 1646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:01.322410 1646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40631,127.11.81.253:32795,127.11.81.252:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.81.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:01.323952 1646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:01.325452 1646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:09:01.326457 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 3
I20260620 07:09:01.327669 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.327111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38282)
0620 07:09:01.327405 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.327504 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 07:09:01.328823 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.328396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38284)
0620 07:09:01.328610 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.328694 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 07:09:01.329064 1565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45061: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45061: connect: Connection refused (error 111)
W20260620 07:09:01.339330 1651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:01.339354 1652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:01.339545 1654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:01.340541 1646 server_base.cc:1034] running on GCE node
I20260620 07:09:01.341431 1646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:01.343022 1646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:01.344264 1646 hybrid_clock.cc:648] HybridClock initialized: now 1781939341344159 us; error 84 us; skew 500 ppm
I20260620 07:09:01.344702 1646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:01.348115 1646 init.cc:377] Logged in from keytab as kudu/127.11.81.252@KRBTEST.COM (short username kudu)
I20260620 07:09:01.349838 1646 webserver.cc:460] Webserver started at http://127.11.81.252:37355/ using document root <none> and password file <none>
I20260620 07:09:01.350612 1646 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:01.350724 1646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:01.351065 1646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:01.354765 1646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data/instance:
uuid: "12a88a3ac114456092da1e39f75a7522"
format_stamp: "Formatted at 2026-06-20 07:09:01 on dist-test-slave-wvz3"
server_key: "4f6fdc9bd66b04a1646b7adcc2f61dd9"
server_key_iv: "087bf0d423ab574f4f268095cf188ca6"
server_key_version: "encryptionkey@0"
I20260620 07:09:01.355839 1646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal/instance:
uuid: "12a88a3ac114456092da1e39f75a7522"
format_stamp: "Formatted at 2026-06-20 07:09:01 on dist-test-slave-wvz3"
server_key: "4f6fdc9bd66b04a1646b7adcc2f61dd9"
server_key_iv: "087bf0d423ab574f4f268095cf188ca6"
server_key_version: "encryptionkey@0"
I20260620 07:09:01.362371 1646 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260620 07:09:01.366402 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:01.368162 1646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:09:01.368362 1646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "12a88a3ac114456092da1e39f75a7522"
format_stamp: "Formatted at 2026-06-20 07:09:01 on dist-test-slave-wvz3"
server_key: "4f6fdc9bd66b04a1646b7adcc2f61dd9"
server_key_iv: "087bf0d423ab574f4f268095cf188ca6"
server_key_version: "encryptionkey@0"
I20260620 07:09:01.368592 1646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:01.387261 1646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:01.392175 1646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:01.392426 1646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:01.410889 1646 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.252:45061
I20260620 07:09:01.410928 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.252:45061 every 8 connection(s)
I20260620 07:09:01.412272 1646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
I20260620 07:09:01.416308 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:01.418551 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1646
I20260620 07:09:01.418706 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938992575046-11591-0/minicluster-data/master-2/wal/instance
I20260620 07:09:01.419144 11591 external_mini_cluster.cc:1428] Setting key 6545f6b1fc412e8b4e4150f6e8dc37f3
I20260620 07:09:01.424832 1713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:09:01.443773 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.421465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.11.81.252:45061)
0620 07:09:01.422485 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.422499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.422549 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:09:01.424431 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.424457 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.424929 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.425370 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.427230 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.428929 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.430490 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.431233 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.431456 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.435981 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.435991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.436006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.436040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.439186 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.440133 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.440140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.440148 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.440318 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.440720 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.440724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.440726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.441333 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.441647 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.442145 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.442578 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":777,"thread_start_us":527,"threads_started":1}
I20260620 07:09:01.443864 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.428075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.11.81.254:40631)
0620 07:09:01.428685 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.428691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.429228 (+ 537us) server_negotiation.cc:407] Connection header received
0620 07:09:01.429408 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.429413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.429496 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.429609 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.430923 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.431748 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.433403 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.434024 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.434153 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.437995 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.438001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.438004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.438042 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.440881 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.441641 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.441645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.441648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.441781 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.442159 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.442163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.442165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.442481 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.442620 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.443088 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.443343 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260620 07:09:01.443863 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.428096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40631 (local address 127.0.0.1:36996)
0620 07:09:01.429064 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.429120 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.429284 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.429718 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.429724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.430410 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.430729 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.430744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.431907 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.431913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.433166 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.433180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.434278 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.434287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.434447 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.435092 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.435125 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.437741 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.441034 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.441046 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.441054 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.441495 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.441899 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.441904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.441908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.442045 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.442592 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.442599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.442792 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.442959 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.443329 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":1,"thread_start_us":553,"threads_started":1}
I20260620 07:09:01.445595 1713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:09:01.461663 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.446744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.11.81.253:32795)
0620 07:09:01.446935 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.446941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.446960 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:09:01.447021 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.447026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.447106 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.447204 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.448413 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.449517 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.451445 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.452106 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.452258 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.455781 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.455787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.455791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.455829 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.458789 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.459611 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.459616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.459619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.459777 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.460228 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.460234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.460237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.460607 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.460761 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.461178 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.461400 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:09:01.461661 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.446444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:32795 (local address 127.0.0.1:59078)
0620 07:09:01.446620 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.446638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.446751 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.447539 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.447543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.448070 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.448223 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.448232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.449667 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.449672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.451213 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.451242 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.452293 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.452298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.452485 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.453082 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.453107 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.455548 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.458972 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.458977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.458980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.459468 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.459932 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.459936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.459939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.460094 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.460745 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.460749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.460924 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.461038 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.461300 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:09:01.462677 1713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:01.464471 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.447146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.11.81.254:40631)
0620 07:09:01.447363 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.447368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.447462 (+ 94us) server_negotiation.cc:407] Connection header received
0620 07:09:01.447649 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.447654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.447770 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.447880 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.449545 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.450821 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.452686 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.453234 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.453383 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.458025 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.458032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.458036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.458083 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.461153 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.462125 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.462130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.462133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.462371 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.462819 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.462823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.462825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.463193 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.463536 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.463955 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.464147 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939341, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:01.477270 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.463852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.11.81.252:45061)
0620 07:09:01.464001 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.464006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.464024 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:09:01.464072 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.464076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.464143 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.464238 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.465055 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.465858 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.467512 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.467958 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.468066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.471846 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.471853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.471858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.471895 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.474498 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.475228 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.475234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.475237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.475448 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.475821 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.475826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.475829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.476121 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.476261 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.476711 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.476939 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:09:01.477274 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.463502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38302)
0620 07:09:01.463697 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.463717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.463833 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.464294 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.464300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.464763 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.464912 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.464921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.465980 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.465985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.467238 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.467256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.468132 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.468139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.468368 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.469002 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.469019 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.471621 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.474663 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.474667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.474670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.475074 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.475559 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.475564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.475568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.475699 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.476239 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.476242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.476444 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.476563 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.476798 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:09:01.483946 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.466635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.11.81.253:32795)
0620 07:09:01.466836 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.466841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.466856 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:09:01.466900 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.466920 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.466982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.467059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.468635 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.469395 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.470900 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.471570 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.471710 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.476238 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.476246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.476251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.476296 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.479719 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.480774 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.480780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.480785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.480982 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.481480 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.481485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.481488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.482001 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.482295 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.483334 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.483594 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:09:01.485510 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: Bootstrap starting.
I20260620 07:09:01.489097 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:01.489143 1713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:01.490213 1713 log.cc:824] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:01.493636 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: No bootstrap required, opened a new log
I20260620 07:09:01.491921 1540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006
I20260620 07:09:01.504731 1713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.505242 1713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a88a3ac114456092da1e39f75a7522, State: Initialized, Role: FOLLOWER
I20260620 07:09:01.505766 1713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.506280 1565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 4
I20260620 07:09:01.506990 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.507443 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.508251 1713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
W20260620 07:09:01.518682 1731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:09:01.518833 1731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:09:01.516541 1646 master_runner.cc:186] Error getting master registration for 127.11.81.254:40631: OK, instance_id { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" instance_seqno: 1781939340363449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:09:01.528347 1646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:09:01.530925 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.507090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.11.81.252:45061)
0620 07:09:01.507353 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.507359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.507519 (+ 160us) server_negotiation.cc:407] Connection header received
0620 07:09:01.507638 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.507643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.507724 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.507817 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.508714 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.509556 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.511342 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.511998 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.512146 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.524033 (+ 11887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.524041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.524046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.524096 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.527891 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.528833 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.528840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.528844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.528992 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.529402 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.529407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.529410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.529768 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.529931 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.530359 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.530570 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 07:09:01.530925 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.507012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38314)
0620 07:09:01.507327 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.507348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.507508 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.507946 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.507951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.508358 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.508558 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.508569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.509745 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.509751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.511116 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.511136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.512184 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.512191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.512429 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.513092 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.513115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.523753 (+ 10638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.528104 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.528111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.528116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.528693 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.529118 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.529124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.529128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.529273 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.529903 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.529908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.530097 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.530221 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.530528 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 07:09:01.537405 1565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: Bootstrap starting.
I20260620 07:09:01.541801 1565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:01.543272 1565 log.cc:824] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:01.547101 1565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: No bootstrap required, opened a new log
I20260620 07:09:01.555248 1565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.555787 1565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b498dd542224a3399f1dbd7fc3dd2e4, State: Initialized, Role: FOLLOWER
I20260620 07:09:01.556336 1565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.557430 1734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.557863 1734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.558835 1565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:01.558894 1540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:09:01.559091 1540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:01.562209 1540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.563926 1540 leader_election.cc:288] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [CANDIDATE]: Term 1 election: Requested vote from peers d98238f707c848628a395bb7a78e4e17 (127.11.81.253:32795), 12a88a3ac114456092da1e39f75a7522 (127.11.81.252:45061)
I20260620 07:09:01.564765 1688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12a88a3ac114456092da1e39f75a7522"
I20260620 07:09:01.565336 1688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:01.569449 1688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b498dd542224a3399f1dbd7fc3dd2e4 in term 1.
I20260620 07:09:01.570266 1516 leader_election.cc:302] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b498dd542224a3399f1dbd7fc3dd2e4, 12a88a3ac114456092da1e39f75a7522; no voters:
I20260620 07:09:01.570744 1734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:01.583411 1734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b498dd542224a3399f1dbd7fc3dd2e4, State: Running, Role: LEADER
I20260620 07:09:01.584326 1734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.592499 1743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b498dd542224a3399f1dbd7fc3dd2e4. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.593132 1743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:01.599521 1751 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:09:01.600672 1752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 07:09:01.603250 1519 tablet.cc:2369] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:01.609543 1639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } attempt: 5
I20260620 07:09:01.604529 1498 master_runner.cc:186] Error getting master registration for 127.11.81.253:32795: OK, instance_id { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" instance_seqno: 1781939340878038 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:09:01.615581 1498 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 07:09:01 dist-test-slave-wvz3 krb5kdc[1485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939340, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
I20260620 07:09:01.623167 1752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:01.631882 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.610212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45061 (local address 127.0.0.1:38330)
0620 07:09:01.610732 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:01.610753 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:01.610874 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:01.611919 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:01.611924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:01.612440 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:01.612700 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.612714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.614363 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.614369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.615882 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:01.615900 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.616948 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.616955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:01.617214 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.617876 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:01.617902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:01.620945 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:01.627700 (+ 6755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.627707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.627712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.628333 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.629118 (+ 785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:01.629124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:01.629127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:01.629312 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:01.630558 (+ 1246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:01.630608 (+ 50us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:01.630814 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 07:09:01.630967 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:09:01.631407 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260620 07:09:01.633670 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.611119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.11.81.252:45061)
0620 07:09:01.611384 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.611390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.611413 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:09:01.611479 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.611488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.611578 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.611685 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.612950 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.614145 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.616037 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.616769 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.616938 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.621204 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.621211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.621217 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.621261 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.627450 (+ 6189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.628598 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.628604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.628608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.628910 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.629514 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.629546 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.629549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.630183 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.632310 (+ 2127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.632587 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.633022 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:09:01.639542 1639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: Bootstrap starting.
I20260620 07:09:01.644304 1639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: Neither blocks nor log segments found. Creating new log.
W20260620 07:09:01.644645 1667 tablet.cc:2369] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:01.656842 1639 log.cc:824] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:01.658972 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.624933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.11.81.254:40631)
0620 07:09:01.625134 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.625141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.632355 (+ 7214us) server_negotiation.cc:407] Connection header received
0620 07:09:01.632605 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.632611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.632694 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.632769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.638620 (+ 5851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.639835 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.643785 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.644491 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.644606 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.649807 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.649816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.649821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.649881 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.653744 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.655069 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.655075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.655082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.655249 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.656043 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.656048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.656052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.656506 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.656775 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.657834 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.658247 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:09:01.663977 1639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: No bootstrap required, opened a new log
I20260620 07:09:01.666072 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.612228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.11.81.252:45061)
0620 07:09:01.624783 (+ 12555us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.624789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.628845 (+ 4056us) server_negotiation.cc:407] Connection header received
0620 07:09:01.629362 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.629369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.629448 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.629535 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.634964 (+ 5429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.636337 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.641721 (+ 5384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.642446 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.642598 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.654528 (+ 11930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.654541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.654546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.654598 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.658179 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.659334 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.659340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.659343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.659529 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.664068 (+ 4539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.664074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.664077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.664515 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.664759 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.665372 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.665648 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11518,"thread_start_us":11297,"threads_started":1}
I20260620 07:09:01.669253 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.605857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.11.81.253:32795)
0620 07:09:01.606050 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.606057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.624607 (+ 18550us) server_negotiation.cc:407] Connection header received
0620 07:09:01.625211 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.625218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.625305 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.625399 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.639645 (+ 14246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.640746 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.642692 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.643358 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.643467 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.659915 (+ 16448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:01.659922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:01.659927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:01.659973 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:01.663714 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.664991 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.664998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.665022 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.665286 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:01.666001 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:01.666035 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:01.666038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:01.666715 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:01.667111 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.668214 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.668707 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:09:01.669570 1688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [term 1 FOLLOWER]: Refusing update from remote peer 0b498dd542224a3399f1dbd7fc3dd2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:01.671275 1734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:01.684264 1639 raft_consensus.cc:348] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.685071 1639 raft_consensus.cc:729] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98238f707c848628a395bb7a78e4e17, State: Initialized, Role: FOLLOWER
I20260620 07:09:01.685979 1639 consensus_queue.cc:260] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } }
I20260620 07:09:01.697077 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b498dd542224a3399f1dbd7fc3dd2e4. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.697453 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.699900 1762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.700484 1762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.702414 1614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d98238f707c848628a395bb7a78e4e17"
I20260620 07:09:01.702891 1614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:01.706247 1614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b498dd542224a3399f1dbd7fc3dd2e4 in term 1.
I20260620 07:09:01.706646 1639 sys_catalog.cc:564] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:01.707903 1613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [term 1 FOLLOWER]: Refusing update from remote peer 0b498dd542224a3399f1dbd7fc3dd2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:01.710021 1743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:01.711540 1752 catalog_manager.cc:1347] Generated new cluster ID: 17bf75daffbb4ddcad2836b20aa47781
I20260620 07:09:01.711649 1752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:01.711923 1734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.712255 1734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:01.736268 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.736652 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.746673 1752 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:01.748548 1752 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:01.766310 1752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4: Generated new TSK 0
I20260620 07:09:01.767588 1752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:01.780303 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.780836 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.780349 1762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b498dd542224a3399f1dbd7fc3dd2e4. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.781283 1762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:01.781622 1743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b498dd542224a3399f1dbd7fc3dd2e4" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40631 } } peers { permanent_uuid: "d98238f707c848628a395bb7a78e4e17" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 32795 } } peers { permanent_uuid: "12a88a3ac114456092da1e39f75a7522" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45061 } } }
I20260620 07:09:01.781888 1743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b498dd542224a3399f1dbd7fc3dd2e4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:01.910672 1784 catalog_manager.cc:1259] Loaded cluster ID: 17bf75daffbb4ddcad2836b20aa47781
I20260620 07:09:01.910809 1784 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: loading cluster ID for follower catalog manager: success
W20260620 07:09:01.913467 1643 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:01.881969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.11.81.253:32795)
0620 07:09:01.882201 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.882208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.911544 (+ 29336us) server_negotiation.cc:407] Connection header received
0620 07:09:01.911899 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.911906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.911994 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.912064 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.913209 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35416: BlockingRecv error: recv got EOF from 127.0.0.1:35416 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:09:01.914945 1784 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: acquiring CA information for follower catalog manager: success
W20260620 07:09:01.917732 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:01.899765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.11.81.252:45061)
0620 07:09:01.899965 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.899972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.915755 (+ 15783us) server_negotiation.cc:407] Connection header received
0620 07:09:01.916212 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.916219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.916311 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.916382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.917488 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:09:01.917795 1784 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d98238f707c848628a395bb7a78e4e17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:09:01.923003 1717 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:01.897194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.11.81.254:40631)
0620 07:09:01.897475 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.897481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.914754 (+ 17273us) server_negotiation.cc:407] Connection header received
0620 07:09:01.921244 (+ 6490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.921252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.921360 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.921448 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:01.922746 (+ 1298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35188: BlockingRecv error: recv got EOF from 127.0.0.1:35188 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:09:01.991253 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:01.965070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.11.81.253:32795)
0620 07:09:01.965285 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:01.965292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:01.974111 (+ 8819us) server_negotiation.cc:407] Connection header received
0620 07:09:01.974600 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:01.974608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:01.974854 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:01.975006 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:01.975795 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.976928 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.987759 (+ 10831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:01.988468 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:01.988610 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:01.989571 (+ 961us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:01.990317 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:01.990478 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:01.990751 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 07:09:01.990868 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 07:09:02.039741 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.036977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.11.81.252:45061)
0620 07:09:02.037173 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.037181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.037695 (+ 514us) server_negotiation.cc:407] Connection header received
0620 07:09:02.038052 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.038059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.038211 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.038298 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.039464 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37372: BlockingRecv error: recv got EOF from 127.0.0.1:37372 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:09:02.115199 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.097551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.11.81.253:32795)
0620 07:09:02.097794 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.097802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.107978 (+ 10176us) server_negotiation.cc:407] Connection header received
0620 07:09:02.108198 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.108204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.108320 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.108418 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.109114 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.110281 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.112716 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.113444 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.113569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.114262 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.114449 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.114520 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.114753 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.114851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 07:09:02.156513 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.146746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.11.81.252:45061)
0620 07:09:02.146929 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.146935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.154413 (+ 7478us) server_negotiation.cc:407] Connection header received
0620 07:09:02.154911 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.154918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.155033 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.155107 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.156271 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37384: BlockingRecv error: recv got EOF from 127.0.0.1:37384 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:09:02.215435 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.198175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.11.81.253:32795)
0620 07:09:02.198378 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.198385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.209291 (+ 10906us) server_negotiation.cc:407] Connection header received
0620 07:09:02.209826 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.209833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.209923 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.209994 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.210777 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.211946 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.213471 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.213927 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.214049 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.214501 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.214646 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.214729 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.214905 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.215047 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 07:09:02.228138 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.225930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.11.81.252:45061)
0620 07:09:02.226135 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.226141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.226645 (+ 504us) server_negotiation.cc:407] Connection header received
0620 07:09:02.226835 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.226841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.226925 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.227009 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.227900 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37386: BlockingRecv error: recv got EOF from 127.0.0.1:37386 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:09:02.247866 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.241842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.11.81.253:32795)
0620 07:09:02.242103 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.242109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.242653 (+ 544us) server_negotiation.cc:407] Connection header received
0620 07:09:02.242852 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.242857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.242939 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.243048 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.243642 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.244546 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.246030 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.246549 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.246674 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.247104 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.247219 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.247337 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.247488 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.247583 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 07:09:02.261232 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.259011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.11.81.252:45061)
0620 07:09:02.259422 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.259429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.259607 (+ 178us) server_negotiation.cc:407] Connection header received
0620 07:09:02.259905 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.259915 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.260048 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.260179 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.260999 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37398: BlockingRecv error: recv got EOF from 127.0.0.1:37398 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 07:09:02.279979 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.273617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.11.81.253:32795)
0620 07:09:02.273836 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.273844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.274057 (+ 213us) server_negotiation.cc:407] Connection header received
0620 07:09:02.274234 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.274240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.274362 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.274486 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.274965 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.276159 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.277713 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.278423 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.278577 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.279080 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.279247 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.279405 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.279542 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.279639 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 07:09:02.288769 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.286563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.11.81.252:45061)
0620 07:09:02.286798 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.286805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.287177 (+ 372us) server_negotiation.cc:407] Connection header received
0620 07:09:02.287373 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.287379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.287481 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.287612 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.288481 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37402: BlockingRecv error: recv got EOF from 127.0.0.1:37402 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:09:02.314102 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.308162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.11.81.253:32795)
0620 07:09:02.308374 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.308380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.308726 (+ 346us) server_negotiation.cc:407] Connection header received
0620 07:09:02.308913 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.308919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.309016 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.309123 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.309636 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.310816 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.312277 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.312821 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.312958 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.313396 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.313510 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.313587 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.313743 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.313838 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:09:02.321950 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.319758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.11.81.252:45061)
0620 07:09:02.319951 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.319958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.320546 (+ 588us) server_negotiation.cc:407] Connection header received
0620 07:09:02.320687 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.320692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.320792 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.320909 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.321725 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37410: BlockingRecv error: recv got EOF from 127.0.0.1:37410 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:09:02.364754 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.358335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.11.81.253:32795)
0620 07:09:02.358550 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.358557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.358812 (+ 255us) server_negotiation.cc:407] Connection header received
0620 07:09:02.359018 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.359024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.359113 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.359223 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.359782 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.360911 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.362461 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.363223 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.363414 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.363878 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.364034 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.364123 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.364309 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.364403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 07:09:02.390090 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.381696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.11.81.252:45061)
0620 07:09:02.381877 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.381883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.388373 (+ 6490us) server_negotiation.cc:407] Connection header received
0620 07:09:02.388578 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.388584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.388711 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.388849 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.389823 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37422: BlockingRecv error: recv got EOF from 127.0.0.1:37422 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:09:02.463538 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.457795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.11.81.253:32795)
0620 07:09:02.458002 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.458007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.458512 (+ 505us) server_negotiation.cc:407] Connection header received
0620 07:09:02.458702 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.458712 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.458791 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.458901 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.459542 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.460451 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.461767 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.462179 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.462292 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.462699 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.462820 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.462895 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.463058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.463165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:09:02.471441 1755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:02.469260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.11.81.252:45061)
0620 07:09:02.469476 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.469481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.469854 (+ 373us) server_negotiation.cc:407] Connection header received
0620 07:09:02.470012 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.470016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.470094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.470195 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:02.471120 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37424: BlockingRecv error: recv got EOF from 127.0.0.1:37424 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 07:09:02.521145 1731 catalog_manager.cc:1259] Loaded cluster ID: 17bf75daffbb4ddcad2836b20aa47781
I20260620 07:09:02.521291 1731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: loading cluster ID for follower catalog manager: success
I20260620 07:09:02.524149 1731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: acquiring CA information for follower catalog manager: success
I20260620 07:09:02.526273 1731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12a88a3ac114456092da1e39f75a7522: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:09:02.608703 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.603317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.11.81.253:32795)
0620 07:09:02.603488 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.603502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.603756 (+ 254us) server_negotiation.cc:407] Connection header received
0620 07:09:02.603925 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.603929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.603998 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.604086 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.604527 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.605332 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.606782 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.607258 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.607402 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.607910 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.608037 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.608112 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.608351 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.608427 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:09:02.621680 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.614916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.11.81.252:45061)
0620 07:09:02.615158 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.615165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.615391 (+ 226us) server_negotiation.cc:407] Connection header received
0620 07:09:02.615704 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.615708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.615881 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.616004 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.616590 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.617406 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.618914 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.619468 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.619581 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.620231 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.620852 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.620973 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.621145 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.621228 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 07:09:02.639842 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.628956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.11.81.254:40631)
0620 07:09:02.632444 (+ 3488us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.632450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.632877 (+ 427us) server_negotiation.cc:407] Connection header received
0620 07:09:02.633121 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.633125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.633291 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.633463 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.634331 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.635067 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.637290 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.637720 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.637831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.638480 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.638964 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.639077 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.639200 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.639333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3349,"thread_start_us":3242,"threads_started":1}
I20260620 07:09:02.640946 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.629285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.11.81.253:32795)
0620 07:09:02.629738 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.629746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.634214 (+ 4468us) server_negotiation.cc:407] Connection header received
0620 07:09:02.635342 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.635349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.635453 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.635547 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.636198 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.637146 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.638950 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.639591 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.639706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.640198 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.640344 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.640427 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.640591 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.640681 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260620 07:09:02.640974 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:02.629489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.11.81.252:45061)
0620 07:09:02.629929 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:02.629935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:02.634553 (+ 4618us) server_negotiation.cc:407] Connection header received
0620 07:09:02.634701 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:02.634705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:02.634774 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:02.634850 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:02.635489 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.636432 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.638439 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:02.639080 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:02.639197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:02.639894 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:02.640049 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:02.640134 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:02.640632 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:09:02.640711 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260620 07:09:02.645380 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1498
I20260620 07:09:02.646728 1560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:02.764108 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1498
I20260620 07:09:02.782347 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1572
I20260620 07:09:02.783538 1634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:02.890190 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1572
I20260620 07:09:02.906745 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1646
I20260620 07:09:02.907927 1708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:03.015705 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1646
2026-06-20T07:09:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4439 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:03 dist-test-slave-wvz3 krb5kdc[1896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:03 dist-test-slave-wvz3 krb5kdc[1896](info): set up 2 sockets
Jun 20 07:09:03 dist-test-slave-wvz3 krb5kdc[1896](info): commencing operation
krb5kdc: starting...
W20260620 07:09:04.056020 11591 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/test-user.keytab.
Jun 20 07:09:04 dist-test-slave-wvz3 krb5kdc[1896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.81.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
Entry for principal kudu/127.11.81.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab.
I20260620 07:09:04.322209 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40491
--webserver_interface=127.11.81.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:04.660576 1909 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:04.660929 1909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:04.661029 1909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:04.661140 1909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:04.669082 1909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:04.669206 1909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:04.669266 1909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:04.669312 1909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:04.669394 1909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:04.679895 1909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.254.keytab
--principal=kudu/127.11.81.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.254:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.81.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:04.681300 1909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:04.682781 1909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:04.695478 1915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:04.695458 1917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:04.695691 1914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:04.696378 1909 server_base.cc:1034] running on GCE node
I20260620 07:09:04.697218 1909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:04.698853 1909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:04.700135 1909 hybrid_clock.cc:648] HybridClock initialized: now 1781939344700065 us; error 35 us; skew 500 ppm
I20260620 07:09:04.700572 1909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:04 dist-test-slave-wvz3 krb5kdc[1896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:04.704177 1909 init.cc:377] Logged in from keytab as kudu/127.11.81.254@KRBTEST.COM (short username kudu)
I20260620 07:09:04.705904 1909 webserver.cc:460] Webserver started at http://127.11.81.254:45591/ using document root <none> and password file <none>
I20260620 07:09:04.706636 1909 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:04.706748 1909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:04.707073 1909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:04.710283 1909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data/instance:
uuid: "241ebb722c364e53ad6da6d7ca07d010"
format_stamp: "Formatted at 2026-06-20 07:09:04 on dist-test-slave-wvz3"
server_key: "ed5c4818cc4b60573935164556c868cc"
server_key_iv: "da4f5b6c0a4d1857b086757091c5ed1a"
server_key_version: "encryptionkey@0"
I20260620 07:09:04.711150 1909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance:
uuid: "241ebb722c364e53ad6da6d7ca07d010"
format_stamp: "Formatted at 2026-06-20 07:09:04 on dist-test-slave-wvz3"
server_key: "ed5c4818cc4b60573935164556c868cc"
server_key_iv: "da4f5b6c0a4d1857b086757091c5ed1a"
server_key_version: "encryptionkey@0"
I20260620 07:09:04.716989 1909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:09:04.720999 1924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:04.722549 1909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 07:09:04.722730 1909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal
uuid: "241ebb722c364e53ad6da6d7ca07d010"
format_stamp: "Formatted at 2026-06-20 07:09:04 on dist-test-slave-wvz3"
server_key: "ed5c4818cc4b60573935164556c868cc"
server_key_iv: "da4f5b6c0a4d1857b086757091c5ed1a"
server_key_version: "encryptionkey@0"
I20260620 07:09:04.722954 1909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:04.744915 1909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:04.749730 1909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:04.750017 1909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:04.769335 1909 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.254:40491
I20260620 07:09:04.769343 1975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.254:40491 every 8 connection(s)
I20260620 07:09:04.770684 1909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:04.774374 1976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:04.778069 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1909
I20260620 07:09:04.778261 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-0/wal/instance
I20260620 07:09:04.778793 11591 external_mini_cluster.cc:1428] Setting key c7766232e6614a7d131f3c6f7ce242e6
I20260620 07:09:04.782723 1976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:04 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.81.253@KRBTEST.COM; defaulting to no policy
I20260620 07:09:04.806506 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.786900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40491 (local address 127.0.0.1:58954)
0620 07:09:04.787766 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.787823 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:04.789889 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:04.790786 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:04.790793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:04.791598 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:04.791962 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:04.791981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:04.793533 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:04.793537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:04.794642 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:04.794655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:04.795627 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:04.795635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:04.795803 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:04.796427 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:04.796451 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:04.798632 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:04.802009 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:04.802021 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:04.802035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:04.802463 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:04.802876 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:04.802881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:04.802884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:04.803038 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:04.803869 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:04.803877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:04.804104 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 07:09:04.804347 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 07:09:04.804985 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":135,"thread_start_us":361,"threads_started":1}
I20260620 07:09:04.806528 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.786901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.11.81.254:40491)
0620 07:09:04.788598 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:04.788607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:04.788656 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:09:04.790303 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:04.790309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:04.790571 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:04.790746 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:04.792164 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:04.793383 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:04.794789 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:04.795404 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:04.795588 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:04.798860 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:04.798869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:04.798882 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:04.798914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:04.801859 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:04.802597 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:04.802604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:04.802606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:04.802768 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:04.803141 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:04.803144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:04.803147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:04.803722 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:04.803934 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:04.804483 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 07:09:04.804842 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1271,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.253@KRBTEST.COM" created.
I20260620 07:09:04.810216 1976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:04.811538 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.811043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33594)
0620 07:09:04.811237 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.811405 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 07:09:04.811820 1927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
I20260620 07:09:04.813508 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.813117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33602)
0620 07:09:04.813268 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.813342 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:09:04.813695 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42405: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
Entry for principal kudu/127.11.81.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab.
I20260620 07:09:04.838608 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:42405
--webserver_interface=127.11.81.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:09:04.863059 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } attempt: 1
I20260620 07:09:04.864362 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.863795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33614)
0620 07:09:04.864015 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.864195 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:09:04.865518 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.865112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33628)
0620 07:09:04.865265 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.865355 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:09:04.865759 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42405: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
I20260620 07:09:04.934056 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } attempt: 2
I20260620 07:09:04.935443 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.934725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33644)
0620 07:09:04.935047 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.935193 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 07:09:04.936596 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:04.936159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33646)
0620 07:09:04.936325 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:04.936420 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:09:04.936854 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42405: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
I20260620 07:09:05.028118 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } attempt: 3
I20260620 07:09:05.029287 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.028747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33652)
0620 07:09:05.028980 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.029115 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:09:05.030678 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.030024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33668)
0620 07:09:05.030347 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.030486 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:09:05.030938 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42405: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
W20260620 07:09:05.172849 1983 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:05.173189 1983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:05.173260 1983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:05.173353 1983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:09:05.174206 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } attempt: 4
I20260620 07:09:05.175560 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.174898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33670)
0620 07:09:05.175227 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.175381 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 07:09:05.176805 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.176359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33676)
0620 07:09:05.176519 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.176627 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:09:05.177068 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.253:42405: Network error: Client connection negotiation failed: client connection to 127.11.81.253:42405: connect: Connection refused (error 111)
W20260620 07:09:05.181638 1983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:05.181748 1983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:05.181805 1983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:05.181875 1983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:05.181947 1983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:05.192662 1983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.253.keytab
--principal=kudu/127.11.81.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.253:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.81.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:05.194027 1983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:05.195436 1983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:05.207834 1989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:05.207862 1991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:05.208009 1988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:05.209110 1983 server_base.cc:1034] running on GCE node
I20260620 07:09:05.210045 1983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:05.211836 1983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:05.213114 1983 hybrid_clock.cc:648] HybridClock initialized: now 1781939345213043 us; error 41 us; skew 500 ppm
I20260620 07:09:05.213573 1983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:05.217221 1983 init.cc:377] Logged in from keytab as kudu/127.11.81.253@KRBTEST.COM (short username kudu)
I20260620 07:09:05.219034 1983 webserver.cc:460] Webserver started at http://127.11.81.253:36775/ using document root <none> and password file <none>
I20260620 07:09:05.219835 1983 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:05.219938 1983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:05.220373 1983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:05.223631 1983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data/instance:
uuid: "ccb97a8e80994c019fb230ea55813d98"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "d64fbeb260d36ee767b153ce5ef4d9f0"
server_key_iv: "174d2ed0a7a4360299ab32e12d386110"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.224488 1983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal/instance:
uuid: "ccb97a8e80994c019fb230ea55813d98"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "d64fbeb260d36ee767b153ce5ef4d9f0"
server_key_iv: "174d2ed0a7a4360299ab32e12d386110"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.230247 1983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:05.234032 1998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:05.235749 1983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:09:05.235958 1983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal
uuid: "ccb97a8e80994c019fb230ea55813d98"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "d64fbeb260d36ee767b153ce5ef4d9f0"
server_key_iv: "174d2ed0a7a4360299ab32e12d386110"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.236193 1983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:05.255271 1983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:05.259976 1983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:05.260262 1983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:05.278673 1983 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.253:42405
I20260620 07:09:05.278680 2049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.253:42405 every 8 connection(s)
I20260620 07:09:05.280144 1983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/data/info.pb
I20260620 07:09:05.284080 2050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:05.284597 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 1983
I20260620 07:09:05.284793 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-1/wal/instance
I20260620 07:09:05.285377 11591 external_mini_cluster.cc:1428] Setting key fc6594984af944cd4d9b79e474def3da
I20260620 07:09:05.293380 2050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.81.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.81.252@KRBTEST.COM" created.
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:09:05.322018 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.296873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.11.81.254:40491)
0620 07:09:05.297076 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.297080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.298709 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 07:09:05.300801 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.300807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.300902 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.300990 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.302633 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.303367 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.304939 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.305436 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.305552 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.315858 (+ 10306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.315864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.315867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.315906 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.318717 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.319624 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.319628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.319631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.319766 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.320215 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.320219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.320222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.320528 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.320667 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.321442 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.321680 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 07:09:05.323051 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.297648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40491 (local address 127.0.0.1:58960)
0620 07:09:05.298521 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.298563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.300490 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.301262 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.301272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.301846 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.302439 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.302460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.303529 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.303533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.304764 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.304779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.305674 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.305682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.305930 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.312795 (+ 6865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.312837 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.315547 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.318917 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.318926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.318942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.319477 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.319888 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.319892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.319894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.320090 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.320668 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.320677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.321053 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.321310 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.321874 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":350,"threads_started":1}
I20260620 07:09:05.324941 2050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:05.340013 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.325815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33680)
0620 07:09:05.326094 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.326109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.326203 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.327388 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.327393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.327872 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.328026 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.328036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.329456 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.329461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.330309 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.330321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.331184 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.331190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.331383 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.332023 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.332043 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.334117 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.337440 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.337446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.337450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.337798 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.338171 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.338176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.338180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.338330 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.338917 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.338921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.339063 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.339212 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.339531 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 07:09:05.340068 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.326028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.11.81.253:42405)
0620 07:09:05.326784 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.326791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.326817 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:09:05.326866 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.326873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.327125 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.327274 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.328210 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.329324 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.330437 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.331020 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.331133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.334330 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.334344 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.334359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.334389 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.337307 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.337935 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.337944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.337946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.338076 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.338423 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.338428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.338431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.338802 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.338993 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.339432 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.339663 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":442,"threads_started":1}
I20260620 07:09:05.342710 2050 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:05.343933 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.343432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50658)
0620 07:09:05.343611 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.343766 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:09:05.344210 2000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
Entry for principal kudu/127.11.81.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab.
I20260620 07:09:05.345793 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.345428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50666)
0620 07:09:05.345591 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.345677 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:09:05.345999 2050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.346431 11591 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
/tmp/dist-test-taskOWNMWE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:45357
--webserver_interface=127.11.81.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.81.212:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:09:05.395393 2050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 1
I20260620 07:09:05.396811 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.396195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50674)
0620 07:09:05.396439 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.396584 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 07:09:05.398000 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.397530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50688)
0620 07:09:05.397724 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.397852 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 07:09:05.398265 2050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.466588 2050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 2
I20260620 07:09:05.467790 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.467233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50704)
0620 07:09:05.467474 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.467588 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 07:09:05.468878 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.468479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50712)
0620 07:09:05.468638 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.468727 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:09:05.469105 2050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.476322 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } attempt: 5
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:05.493258 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.477290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.11.81.253:42405)
0620 07:09:05.477536 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.477544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.480350 (+ 2806us) server_negotiation.cc:407] Connection header received
0620 07:09:05.480575 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.480580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.480681 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.480807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.481978 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.483032 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.484354 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.484781 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.484889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.488111 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.488116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.488120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.488154 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.490721 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.491428 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.491433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.491436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.491559 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.491909 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.491913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.491915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.492196 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.492330 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.492719 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.492932 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:09:05.493263 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.477090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33686)
0620 07:09:05.480221 (+ 3131us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.480248 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.480418 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.480972 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.480979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.481534 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.481793 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.481810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.483233 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.483239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.484182 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.484207 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.484939 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.484945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.485150 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.485785 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.485804 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.487911 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.490867 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.490873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.490877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.491279 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.491662 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.491667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.491671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.491804 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.492306 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.492311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.492461 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.492579 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.492898 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:09:05.494534 1976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:09:05.495738 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.495189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50726)
0620 07:09:05.495464 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.495560 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 07:09:05.496977 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.496682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50738)
0620 07:09:05.496811 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.496878 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 07:09:05.497160 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.548434 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 1
I20260620 07:09:05.549539 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.549073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50740)
0620 07:09:05.549271 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.549369 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:09:05.550671 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.550296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50746)
0620 07:09:05.550472 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.550553 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 07:09:05.550907 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.560362 2050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 3
I20260620 07:09:05.561717 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.561112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50748)
0620 07:09:05.561355 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.561493 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 07:09:05.563055 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.562485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50762)
0620 07:09:05.562747 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.562864 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 07:09:05.563278 2050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.619168 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 2
I20260620 07:09:05.620478 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.619969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50772)
0620 07:09:05.620224 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.620317 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:09:05.621482 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.621161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50782)
0620 07:09:05.621293 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.621370 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:09:05.621688 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
W20260620 07:09:05.682862 2057 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:09:05.683216 2057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:05.683326 2057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:05.683442 2057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:05.691516 2057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:05.691618 2057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:05.691674 2057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:05.691723 2057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:05.691787 2057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:05.701994 2057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.81.212:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.81.254:40491,127.11.81.253:42405,127.11.81.252:45357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/krb5kdc/kudu_127.11.81.252.keytab
--principal=kudu/127.11.81.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.81.252:45357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.81.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:05.703361 2057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:05.704667 2057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:09:05.706547 2050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 4
I20260620 07:09:05.707811 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.707232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50796)
0620 07:09:05.707491 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.707597 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 07:09:05.708985 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.708566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50806)
0620 07:09:05.708726 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.708826 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:09:05.709285 2050 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
W20260620 07:09:05.717036 2062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:05.717160 2065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:05.717109 2063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:05.718482 2057 server_base.cc:1034] running on GCE node
I20260620 07:09:05.719380 2057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:05.720909 2057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:05.722141 2057 hybrid_clock.cc:648] HybridClock initialized: now 1781939345722073 us; error 49 us; skew 500 ppm
I20260620 07:09:05.722486 2057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:05.725684 2057 init.cc:377] Logged in from keytab as kudu/127.11.81.252@KRBTEST.COM (short username kudu)
I20260620 07:09:05.727255 2057 webserver.cc:460] Webserver started at http://127.11.81.252:45067/ using document root <none> and password file <none>
I20260620 07:09:05.727901 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 3
I20260620 07:09:05.728053 2057 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:05.728168 2057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:05.728556 2057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:05.729043 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.728533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50810)
0620 07:09:05.728786 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.728873 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:09:05.730154 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.729799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50826)
0620 07:09:05.729965 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.730043 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 07:09:05.730367 1976 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.81.252:45357: Network error: Client connection negotiation failed: client connection to 127.11.81.252:45357: connect: Connection refused (error 111)
I20260620 07:09:05.732038 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data/instance:
uuid: "1dc316bb35314202b34b508c86fbc8c3"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "6ab2c3a862144dd1d3365de4e070b2cc"
server_key_iv: "84da8cc36329a990f4a8fc414cb6d08b"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.732856 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal/instance:
uuid: "1dc316bb35314202b34b508c86fbc8c3"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "6ab2c3a862144dd1d3365de4e070b2cc"
server_key_iv: "84da8cc36329a990f4a8fc414cb6d08b"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.738234 2057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:09:05.741991 2072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:05.743594 2057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:05.743767 2057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data,/tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal
uuid: "1dc316bb35314202b34b508c86fbc8c3"
format_stamp: "Formatted at 2026-06-20 07:09:05 on dist-test-slave-wvz3"
server_key: "6ab2c3a862144dd1d3365de4e070b2cc"
server_key_iv: "84da8cc36329a990f4a8fc414cb6d08b"
server_key_version: "encryptionkey@0"
I20260620 07:09:05.743979 2057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:05.762920 2057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:05.767465 2057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:05.767763 2057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:05.787631 2057 rpc_server.cc:307] RPC server started. Bound to: 127.11.81.252:45357
I20260620 07:09:05.787649 2123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.81.252:45357 every 8 connection(s)
I20260620 07:09:05.789021 2057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/data/info.pb
I20260620 07:09:05.792094 11591 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu as pid 2057
I20260620 07:09:05.792297 11591 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOWNMWE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938992575046-11591-0/minicluster-data/master-2/wal/instance
I20260620 07:09:05.792843 11591 external_mini_cluster.cc:1428] Setting key 4098e982483e67fbf91c77ceca5a98e6
I20260620 07:09:05.793344 2124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:05.803103 2124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
I20260620 07:09:05.817833 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.794681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.11.81.252:45357)
0620 07:09:05.795578 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.795590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.795632 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:09:05.798020 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.798047 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.798515 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.798903 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.800808 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.802162 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.804022 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.804832 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.805001 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.809753 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.809764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.809779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.809828 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.813412 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.814170 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.814179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.814191 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.814366 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.814732 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.814744 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.814748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.815499 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.815799 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.816294 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.816665 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":366,"threads_started":1}
I20260620 07:09:05.822219 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.806413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.254:40491 (local address 127.0.0.1:58964)
0620 07:09:05.807308 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.807365 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.807549 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.807973 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.807981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.808607 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.808939 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.808952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.810225 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.810229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.811497 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.811509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.812274 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.812282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.812452 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.813181 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.813207 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.815560 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.819210 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.819218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.819225 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.819825 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.820301 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.820306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.820310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.820449 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.821100 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.821108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.821416 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.821577 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.821833 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":402,"threads_started":1}
I20260620 07:09:05.822315 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.806490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.11.81.254:40491)
0620 07:09:05.806712 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.806720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.807466 (+ 746us) server_negotiation.cc:407] Connection header received
0620 07:09:05.807706 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.807711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.807782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.807871 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.809106 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.810056 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.811620 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.812062 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.812162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.815800 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.815807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.815812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.815862 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.818873 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.819995 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.820000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.820002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.820160 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.820576 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.820581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.820585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.820934 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.821133 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.821737 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.821974 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:09:05.824092 2124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.254@KRBTEST.COM
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:05.837065 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.820960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.11.81.254:40491)
0620 07:09:05.821630 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.821635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.821656 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:09:05.821700 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.821705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.821770 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.821881 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.823339 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.824351 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.825853 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.826260 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.826426 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.831224 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.831230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.831234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.831271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.834217 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.834986 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.834991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.834995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.835127 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.835563 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.835569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.835571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.835897 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.836155 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.836481 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.836704 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":352,"threads_started":1}
I20260620 07:09:05.839267 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.824919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.253:42405 (local address 127.0.0.1:33702)
0620 07:09:05.825260 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.825279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.825392 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.825749 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.825753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.826123 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.826310 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.826321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.827386 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.827391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.828818 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.828834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.829611 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.829616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.829737 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.831210 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.831229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.833179 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.836312 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.836318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.836323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.836714 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.837147 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.837152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.837156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.837353 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.838117 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.838123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.838335 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.838490 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.838857 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260620 07:09:05.839403 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.824923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.11.81.253:42405)
0620 07:09:05.825108 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.825115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.825362 (+ 247us) server_negotiation.cc:407] Connection header received
0620 07:09:05.825548 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.825552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.825620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.825704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.826448 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.827155 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.828960 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.829454 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.829584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.833400 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.833407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.833412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.833457 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.836165 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.836844 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.836849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.836851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.837030 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.837564 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.837570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.837573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.837940 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.838109 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.838614 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.838857 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:09:05.840369 2124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.252@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.81.253@KRBTEST.COM
I20260620 07:09:05.855178 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.841504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.11.81.252:45357)
0620 07:09:05.841661 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.841665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.841678 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:09:05.841714 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.841717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.841771 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.841858 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.842700 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.843829 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.845377 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.845872 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.846013 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.849391 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.849396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.849400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.849431 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.852525 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.853225 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.853231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.853235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.853362 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.853726 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.853731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.853735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.854041 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.854190 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.854536 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.854767 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:09:05.855334 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.841142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50838)
0620 07:09:05.841308 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.841326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.841429 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.841889 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.841894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.842338 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.842523 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.842536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.843986 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.843993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.845202 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.845217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.846060 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.846067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.846322 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.846832 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.846854 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.849198 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.852666 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.852673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.852677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.853082 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.853466 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.853471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.853475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.853616 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.854155 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.854160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.854306 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.854444 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.854920 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":157}
I20260620 07:09:05.856262 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.838894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.11.81.253:42405)
0620 07:09:05.839965 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.839971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.839989 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:09:05.840038 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.840044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.840120 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.840249 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.842311 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.843226 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.844861 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.845609 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.845742 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.850890 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.850895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.850899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.850945 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.853533 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.854173 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.854178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.854182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.854359 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.854737 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.854742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.854744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.854996 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.855669 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.855809 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.856023 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":8552,"threads_started":1}
I20260620 07:09:05.861133 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: Bootstrap starting.
I20260620 07:09:05.861279 11591 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:05.865007 2124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:05.866541 2124 log.cc:824] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:05.863641 1951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "241ebb722c364e53ad6da6d7ca07d010"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58968
I20260620 07:09:05.869964 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: No bootstrap required, opened a new log
I20260620 07:09:05.881624 2124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.882136 2124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc316bb35314202b34b508c86fbc8c3, State: Initialized, Role: FOLLOWER
I20260620 07:09:05.882714 2124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.883845 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:05.884176 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:05.885063 2124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:09:05.894622 2143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:09:05.894765 2143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:09:05.892720 2057 master_runner.cc:186] Error getting master registration for 127.11.81.254:40491: OK, instance_id { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" instance_seqno: 1781939344763048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:09:05.903847 2057 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:09:05.907598 1976 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 4
Jun 20 07:09:05 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.254@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
I20260620 07:09:05.923734 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.908387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.11.81.252:45357)
0620 07:09:05.908622 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.908629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:05.908900 (+ 271us) server_negotiation.cc:407] Connection header received
0620 07:09:05.909125 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:05.909130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:05.909214 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:05.909315 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:05.910330 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.911514 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.913381 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.913875 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.913970 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.918239 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:05.918244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:05.918247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:05.918284 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:05.921035 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.921751 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.921755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.921758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.921877 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:05.922261 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:05.922265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:05.922267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:05.922603 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:05.922748 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:05.923130 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 07:09:05.923368 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:09:05.923734 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.908382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50850)
0620 07:09:05.908801 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:05.908824 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:05.908982 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:05.909371 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:05.909377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:05.909905 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:05.910145 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.910171 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.911682 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.911689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.913203 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:05.913223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:05.914031 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:05.914039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:05.914286 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:05.914974 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:05.915002 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:05.917999 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:05.921218 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.921224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.921228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.921625 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.921990 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:05.921996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:05.922000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:05.922145 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:05.922715 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:05.922720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:05.922881 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 07:09:05.923004 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:09:05.923357 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260620 07:09:05.928864 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: Bootstrap starting.
I20260620 07:09:05.932174 1976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:05.933495 1976 log.cc:824] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:05.937388 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: No bootstrap required, opened a new log
I20260620 07:09:05.943671 1976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.944031 1976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241ebb722c364e53ad6da6d7ca07d010, State: Initialized, Role: FOLLOWER
I20260620 07:09:05.944485 1976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.945457 2146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:05.945909 2146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:05.946727 1976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:05.946849 1951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:09:05.947022 1951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:05.950938 1951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.952823 1951 leader_election.cc:288] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [CANDIDATE]: Term 1 election: Requested vote from peers ccb97a8e80994c019fb230ea55813d98 (127.11.81.253:42405), 1dc316bb35314202b34b508c86fbc8c3 (127.11.81.252:45357)
I20260620 07:09:05.955102 2099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "241ebb722c364e53ad6da6d7ca07d010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1dc316bb35314202b34b508c86fbc8c3"
I20260620 07:09:05.955899 2099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:05.959757 2099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241ebb722c364e53ad6da6d7ca07d010 in term 1.
I20260620 07:09:05.960796 1926 leader_election.cc:302] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dc316bb35314202b34b508c86fbc8c3, 241ebb722c364e53ad6da6d7ca07d010; no voters:
I20260620 07:09:05.961423 2146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:05.972874 2146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [term 1 LEADER]: Becoming Leader. State: Replica: 241ebb722c364e53ad6da6d7ca07d010, State: Running, Role: LEADER
I20260620 07:09:05.974253 2146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:05.986835 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241ebb722c364e53ad6da6d7ca07d010. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:05.987895 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: This master's current role is: LEADER
W20260620 07:09:06.003059 1930 tablet.cc:2369] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:06.005417 2158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:06.008493 2050 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } attempt: 5
W20260620 07:09:06.023146 2078 tablet.cc:2369] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:06.032985 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.990484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.11.81.253:42405)
0620 07:09:05.990680 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.990687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.000088 (+ 9401us) server_negotiation.cc:407] Connection header received
0620 07:09:06.000437 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.000445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.000539 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.000621 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.003127 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.004323 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.015575 (+ 11252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.016382 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.016521 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.022100 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:06.022108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:06.022114 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:06.022168 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:06.026311 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.029231 (+ 2920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.029239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.029244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.029455 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.029972 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.029977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.029980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.030585 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:06.030987 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.032073 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.032464 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:09:06.037870 2158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 07:09:06 dist-test-slave-wvz3 krb5kdc[1896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.81.253@KRBTEST.COM for kudu/127.11.81.252@KRBTEST.COM
I20260620 07:09:06.049427 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:05.992482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.11.81.252:45357)
0620 07:09:05.992669 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:05.992675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.005474 (+ 12799us) server_negotiation.cc:407] Connection header received
0620 07:09:06.006009 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.006016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.006115 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.006206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.021506 (+ 15300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.022766 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.028706 (+ 5940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.029331 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.029422 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.038693 (+ 9271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:06.038700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:06.038705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:06.038753 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:06.043649 (+ 4896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.045704 (+ 2055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.045712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.045716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.045924 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.046639 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.046644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.046647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.047075 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:06.047419 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.048221 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.049038 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:09:06.054801 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.009187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.81.252:45357 (local address 127.0.0.1:50866)
0620 07:09:06.009601 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:06.009620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:06.009743 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:06.028372 (+ 18629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:06.028377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:06.028869 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:06.029065 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:06.029078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.035295 (+ 6217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.035302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:06.036701 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:06.036718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.037730 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.037736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:06.037960 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.038635 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:06.038660 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:06.044660 (+ 6000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:06.051767 (+ 7107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:06.051774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:06.051777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:06.052337 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:06.052869 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:06.052875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:06.052879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:06.053058 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:06.053727 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:06.053733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:06.053918 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 07:09:06.054085 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 07:09:06.054407 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260620 07:09:06.055109 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.009961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.11.81.252:45357)
0620 07:09:06.028014 (+ 18053us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.028020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.028045 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:09:06.028103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.028108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.028192 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.028308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.032439 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.035116 (+ 2677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.036918 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.037579 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.037698 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.045799 (+ 8101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:06.045807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:06.045813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:06.045859 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:06.051578 (+ 5719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.052539 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.052544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.052548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.052718 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.053179 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.053184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.053191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.053595 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:06.053742 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.054194 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.054435 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17903,"thread_start_us":18501,"threads_started":1}
I20260620 07:09:06.059567 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: Bootstrap starting.
I20260620 07:09:06.064002 2050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:06.065470 2050 log.cc:824] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:06.067710 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.003350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.11.81.254:40491)
0620 07:09:06.003561 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.003567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.020680 (+ 17113us) server_negotiation.cc:407] Connection header received
0620 07:09:06.021584 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.021592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.021719 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.021804 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.030009 (+ 8205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.031212 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.038982 (+ 7770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.048284 (+ 9302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.051833 (+ 3549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.055477 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:06.055485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:06.055490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:06.055536 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:06.063505 (+ 7969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.064553 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.064559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.064563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.064732 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:06.065233 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:06.065238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:06.065242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:06.065653 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:06.066894 (+ 1241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.067048 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.067347 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:09:06.076864 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: No bootstrap required, opened a new log
I20260620 07:09:06.063505 1909 master_runner.cc:186] Error getting master registration for 127.11.81.253:42405: OK, instance_id { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" instance_seqno: 1781939345273447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:09:06.077109 1909 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:09:06.083995 2099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [term 1 FOLLOWER]: Refusing update from remote peer 241ebb722c364e53ad6da6d7ca07d010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:06.085163 2146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:06.090348 2050 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:06.090988 2050 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb97a8e80994c019fb230ea55813d98, State: Initialized, Role: FOLLOWER
I20260620 07:09:06.091589 2050 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } }
I20260620 07:09:06.100934 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.102700 2050 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:06.102660 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:06.103139 2024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:06.107322 2024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [term 1 FOLLOWER]: Refusing update from remote peer 241ebb722c364e53ad6da6d7ca07d010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:06.108191 2025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "241ebb722c364e53ad6da6d7ca07d010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccb97a8e80994c019fb230ea55813d98"
I20260620 07:09:06.112720 2146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:06.113318 2025 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241ebb722c364e53ad6da6d7ca07d010 in term 1.
I20260620 07:09:06.121084 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241ebb722c364e53ad6da6d7ca07d010. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.121429 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:06.130888 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241ebb722c364e53ad6da6d7ca07d010. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.131394 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:06.131868 2152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.132133 2152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:06.136154 2158 catalog_manager.cc:1347] Generated new cluster ID: d656e321990a4fc4ab207fd490ce2d5c
I20260620 07:09:06.136273 2158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:06.139426 2176 mvcc.cc:205] Tried to move back new op lower bound from 7298823561507061760 to 7298823561123610624. Current Snapshot: MvccSnapshot[applied={T|T < 7298823561507061760}]
I20260620 07:09:06.144349 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.144644 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:06.144855 2146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.145215 2146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:06.164031 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "241ebb722c364e53ad6da6d7ca07d010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "241ebb722c364e53ad6da6d7ca07d010" member_type: VOTER last_known_addr { host: "127.11.81.254" port: 40491 } } peers { permanent_uuid: "ccb97a8e80994c019fb230ea55813d98" member_type: VOTER last_known_addr { host: "127.11.81.253" port: 42405 } } peers { permanent_uuid: "1dc316bb35314202b34b508c86fbc8c3" member_type: VOTER last_known_addr { host: "127.11.81.252" port: 45357 } } }
I20260620 07:09:06.164325 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:09:06.179323 2158 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:06.181403 2158 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:06.204813 2158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241ebb722c364e53ad6da6d7ca07d010: Generated new TSK 0
I20260620 07:09:06.206099 2158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:06.247025 2191 catalog_manager.cc:1259] Loaded cluster ID: d656e321990a4fc4ab207fd490ce2d5c
I20260620 07:09:06.247238 2191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: loading cluster ID for follower catalog manager: success
I20260620 07:09:06.251611 2191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: acquiring CA information for follower catalog manager: success
I20260620 07:09:06.254648 2191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccb97a8e80994c019fb230ea55813d98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:09:06.330009 11591 mini_kdc.cc:348] Time spent kdestroy: real 0.106s user 0.001s sys 0.022s
W20260620 07:09:06.381976 2055 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.366501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.11.81.253:42405)
0620 07:09:06.366705 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.366711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.379878 (+ 13167us) server_negotiation.cc:407] Connection header received
0620 07:09:06.380192 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.380197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.380306 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.380392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.381718 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33720: BlockingRecv error: recv got EOF from 127.0.0.1:33720 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 07:09:06.382798 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.368335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.11.81.252:45357)
0620 07:09:06.368499 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.368505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.380990 (+ 12485us) server_negotiation.cc:407] Connection header received
0620 07:09:06.381293 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.381300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.381400 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.381475 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.382578 (+ 1103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50874: BlockingRecv error: recv got EOF from 127.0.0.1:50874 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 07:09:06.384276 2129 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.380132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.11.81.254:40491)
0620 07:09:06.380358 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.380364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.382615 (+ 2251us) server_negotiation.cc:407] Connection header received
0620 07:09:06.382880 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.382886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.382967 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.383037 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.384004 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58984: BlockingRecv error: recv got EOF from 127.0.0.1:58984 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:09:06.429512 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.413347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.11.81.253:42405)
0620 07:09:06.413550 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.413557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.420940 (+ 7383us) server_negotiation.cc:407] Connection header received
0620 07:09:06.421516 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.421523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.421781 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.421923 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.422622 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.423826 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.426178 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.426920 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.427039 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.427936 (+ 897us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.428686 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.428824 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.429070 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.429177 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 07:09:06.478746 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.463648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.11.81.252:45357)
0620 07:09:06.463860 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.463867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.476441 (+ 12574us) server_negotiation.cc:407] Connection header received
0620 07:09:06.477023 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.477031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.477177 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.477276 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.478456 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50882: BlockingRecv error: recv got EOF from 127.0.0.1:50882 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:09:06.523259 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.508402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.11.81.253:42405)
0620 07:09:06.508607 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.508614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.516111 (+ 7497us) server_negotiation.cc:407] Connection header received
0620 07:09:06.516672 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.516679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.516773 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.516840 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.517574 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.518742 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.520827 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.521519 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.521625 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.522316 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.522496 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.522559 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.522786 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.522890 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:09:06.545588 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.540155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.11.81.252:45357)
0620 07:09:06.540389 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.540395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.543919 (+ 3524us) server_negotiation.cc:407] Connection header received
0620 07:09:06.544127 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.544134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.544257 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.544393 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.545346 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50892: BlockingRecv error: recv got EOF from 127.0.0.1:50892 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:09:06.563817 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.557038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.11.81.253:42405)
0620 07:09:06.557235 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.557241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.557691 (+ 450us) server_negotiation.cc:407] Connection header received
0620 07:09:06.557898 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.557903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.557990 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.558090 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.558566 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.559705 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.561657 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.562300 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.562431 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.562934 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.563113 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.563208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.563393 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.563483 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:09:06.575994 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.573460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.11.81.252:45357)
0620 07:09:06.573651 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.573658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.574268 (+ 610us) server_negotiation.cc:407] Connection header received
0620 07:09:06.574563 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.574570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.574681 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.574807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.575753 (+ 946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50904: BlockingRecv error: recv got EOF from 127.0.0.1:50904 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:09:06.590860 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.583529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.11.81.253:42405)
0620 07:09:06.583718 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.583724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.584255 (+ 531us) server_negotiation.cc:407] Connection header received
0620 07:09:06.584464 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.584471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.584577 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.584704 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.585206 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.586490 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.588520 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.589303 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.589450 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.590042 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.590201 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.590325 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.590452 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.590547 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:09:06.601490 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.598114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.11.81.252:45357)
0620 07:09:06.598370 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.598375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.599846 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 07:09:06.600049 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.600055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.600175 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.600351 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.601246 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50906: BlockingRecv error: recv got EOF from 127.0.0.1:50906 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 07:09:06.620297 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.613395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.11.81.253:42405)
0620 07:09:06.613582 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.613588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.614162 (+ 574us) server_negotiation.cc:407] Connection header received
0620 07:09:06.614374 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.614381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.614478 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.614592 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.615022 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.616238 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.618165 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.618776 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.618905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.619480 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.619630 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.619722 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.619876 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.619967 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:09:06.629994 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.627534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.11.81.252:45357)
0620 07:09:06.627720 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.627727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.628298 (+ 571us) server_negotiation.cc:407] Connection header received
0620 07:09:06.628491 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.628496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.628591 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.628693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.629623 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50920: BlockingRecv error: recv got EOF from 127.0.0.1:50920 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:09:06.655112 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.649766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.11.81.253:42405)
0620 07:09:06.649976 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.649982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.650296 (+ 314us) server_negotiation.cc:407] Connection header received
0620 07:09:06.650511 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.650517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.650597 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.650679 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.651113 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.652051 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.653410 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.653883 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.654001 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.654437 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.654552 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.654633 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.654771 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.654859 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 07:09:06.664279 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.662249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.11.81.252:45357)
0620 07:09:06.662452 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.662459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.662774 (+ 315us) server_negotiation.cc:407] Connection header received
0620 07:09:06.662935 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.662940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.663039 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.663161 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.664026 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50936: BlockingRecv error: recv got EOF from 127.0.0.1:50936 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:09:06.706573 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.700226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.11.81.253:42405)
0620 07:09:06.700469 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.700478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.700788 (+ 310us) server_negotiation.cc:407] Connection header received
0620 07:09:06.701004 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.701009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.701130 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.701250 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.701611 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.702779 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.704484 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.705173 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.705313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.705800 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.705944 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.706037 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.706167 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.706258 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 07:09:06.715752 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.713544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.11.81.252:45357)
0620 07:09:06.713739 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.713743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.714196 (+ 453us) server_negotiation.cc:407] Connection header received
0620 07:09:06.714340 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.714345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.714444 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.714549 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.715521 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50952: BlockingRecv error: recv got EOF from 127.0.0.1:50952 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:09:06.789219 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.783150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.11.81.253:42405)
0620 07:09:06.783357 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.783362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.783659 (+ 297us) server_negotiation.cc:407] Connection header received
0620 07:09:06.783843 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.783848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.783946 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.784060 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.784502 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.785656 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.787255 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.787920 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.788051 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.788505 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.788643 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.788737 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.788831 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.788922 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 07:09:06.804610 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:06.796806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.11.81.252:45357)
0620 07:09:06.797027 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.797034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.803151 (+ 6117us) server_negotiation.cc:407] Connection header received
0620 07:09:06.803333 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.803338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.803452 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.803605 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:06.804390 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50968: BlockingRecv error: recv got EOF from 127.0.0.1:50968 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:09:06.896751 2143 catalog_manager.cc:1259] Loaded cluster ID: d656e321990a4fc4ab207fd490ce2d5c
I20260620 07:09:06.896850 2143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: loading cluster ID for follower catalog manager: success
I20260620 07:09:06.899326 2143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: acquiring CA information for follower catalog manager: success
I20260620 07:09:06.901156 2143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1dc316bb35314202b34b508c86fbc8c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:09:06.940992 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.935883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.11.81.253:42405)
0620 07:09:06.936139 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.936146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.936397 (+ 251us) server_negotiation.cc:407] Connection header received
0620 07:09:06.936598 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.936604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.936679 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.936765 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.937193 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.937936 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.939306 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.939768 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.939877 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.940266 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.940390 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.940464 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.940598 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.940678 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 07:09:06.953370 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.946647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.11.81.252:45357)
0620 07:09:06.946851 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.946856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.947069 (+ 213us) server_negotiation.cc:407] Connection header received
0620 07:09:06.947381 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.947386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.947564 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.947682 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.948108 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.948892 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.950529 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.951169 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.951357 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.951932 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.952727 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.952890 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.952941 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.953037 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 07:09:06.973644 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.963013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.11.81.252:45357)
0620 07:09:06.963198 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.963204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.966641 (+ 3437us) server_negotiation.cc:407] Connection header received
0620 07:09:06.966874 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.966880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.967013 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.967131 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.968072 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.969302 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.971493 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.972134 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.972229 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.972888 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.973042 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.973109 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.973341 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.973427 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:09:06.977110 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.962619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.11.81.253:42405)
0620 07:09:06.962846 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.962854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.970582 (+ 7728us) server_negotiation.cc:407] Connection header received
0620 07:09:06.970785 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.970790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.970903 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.970998 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.971880 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.972918 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.975391 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.975922 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.976045 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.976448 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.976558 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.976632 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.976781 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.976860 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:09:06.984145 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:06.962487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.11.81.254:40491)
0620 07:09:06.978920 (+ 16433us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:06.978925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:06.978944 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:09:06.979052 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:06.979056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:06.979229 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:06.979438 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:06.979831 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.980561 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.981955 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:06.982443 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:06.982569 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:06.983079 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:06.983490 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:06.983596 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:06.983733 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 07:09:06.983826 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16321,"thread_start_us":16843,"threads_started":1}
I20260620 07:09:06.989965 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1909
I20260620 07:09:06.991080 1971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:07.102615 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1909
I20260620 07:09:07.120225 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 1983
I20260620 07:09:07.121380 2045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:07.229554 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 1983
I20260620 07:09:07.246690 11591 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu pid 2057
I20260620 07:09:07.247879 2119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:07.354748 11591 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOWNMWE/build/asan/bin/kudu with pid 2057
2026-06-20T07:09:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4338 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8778 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (354744 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST